Jul 14 15:19:01.587648 ip-10-0-128-22 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 14 15:19:01.587658 ip-10-0-128-22 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 14 15:19:01.587665 ip-10-0-128-22 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 14 15:19:01.587887 ip-10-0-128-22 systemd[1]: Failed to start Kubernetes Kubelet. Jul 14 15:19:11.703461 ip-10-0-128-22 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 14 15:19:11.703479 ip-10-0-128-22 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 28a2d3043bc845cdbb38659a8f4ce460 -- Jul 14 15:20:57.037361 ip-10-0-128-22 systemd[1]: Starting Kubernetes Kubelet... Jul 14 15:20:57.489101 ip-10-0-128-22 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 15:20:57.489101 ip-10-0-128-22 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 14 15:20:57.489101 ip-10-0-128-22 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 15:20:57.489101 ip-10-0-128-22 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 14 15:20:57.489101 ip-10-0-128-22 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 15:20:57.489759 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.489167 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 14 15:20:57.494514 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494492 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 15:20:57.494514 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494509 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 15:20:57.494514 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494514 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 15:20:57.494514 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494521 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 15:20:57.494757 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494525 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 15:20:57.494757 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494529 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 15:20:57.494757 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494533 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 15:20:57.494757 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494537 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 15:20:57.494757 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494541 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 15:20:57.494757 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494544 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 15:20:57.494757 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494547 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 15:20:57.494757 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494551 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 15:20:57.494757 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494555 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 15:20:57.494757 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494561 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 15:20:57.494757 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494568 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 15:20:57.494757 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494572 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 15:20:57.494757 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494576 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 15:20:57.494757 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494579 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 15:20:57.494757 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494583 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 15:20:57.494757 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494587 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 15:20:57.494757 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494591 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 15:20:57.494757 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494594 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 15:20:57.494757 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494598 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 15:20:57.494757 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494602 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 15:20:57.495402 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494606 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 15:20:57.495402 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494609 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 15:20:57.495402 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494614 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 15:20:57.495402 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494618 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 15:20:57.495402 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494622 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 15:20:57.495402 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494626 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 15:20:57.495402 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494630 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 15:20:57.495402 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494634 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 15:20:57.495402 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494643 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 15:20:57.495402 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494647 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 15:20:57.495402 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494652 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 15:20:57.495402 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494656 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 15:20:57.495402 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494660 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 15:20:57.495402 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494665 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 15:20:57.495402 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494669 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 15:20:57.495402 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494674 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 15:20:57.495402 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494678 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 15:20:57.495402 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494682 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 15:20:57.495402 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494689 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 15:20:57.495402 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494694 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 15:20:57.495925 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494699 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 15:20:57.495925 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494704 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 15:20:57.495925 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494711 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 15:20:57.495925 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494715 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 14 15:20:57.495925 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494720 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 15:20:57.495925 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494725 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 15:20:57.495925 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494730 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 15:20:57.495925 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494735 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 15:20:57.495925 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494740 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 15:20:57.495925 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494744 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 15:20:57.495925 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494749 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 15:20:57.495925 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494753 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 15:20:57.495925 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494757 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 15:20:57.495925 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494761 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 15:20:57.495925 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494765 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 15:20:57.495925 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494770 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 15:20:57.495925 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494774 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 15:20:57.495925 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494779 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 15:20:57.495925 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494783 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 15:20:57.495925 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494788 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 15:20:57.496768 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494792 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 15:20:57.496768 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494799 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 15:20:57.496768 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494804 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 15:20:57.496768 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494809 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 15:20:57.496768 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494813 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 15:20:57.496768 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494817 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 15:20:57.496768 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494821 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 15:20:57.496768 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494825 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 15:20:57.496768 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494829 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 15:20:57.496768 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494833 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 15:20:57.496768 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494838 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 15:20:57.496768 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494842 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 15:20:57.496768 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494846 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 15:20:57.496768 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494850 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 15:20:57.496768 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494855 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 15:20:57.496768 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494859 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 15:20:57.496768 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494863 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 15:20:57.496768 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494867 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 15:20:57.496768 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494872 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 15:20:57.497589 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494878 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 15:20:57.497589 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494882 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 15:20:57.497589 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494887 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 15:20:57.497589 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494892 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 15:20:57.497589 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494898 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 15:20:57.497589 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494904 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 15:20:57.497589 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494909 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 15:20:57.497589 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494913 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 15:20:57.497589 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494918 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 15:20:57.497589 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494922 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 15:20:57.497589 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494927 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 15:20:57.497589 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494931 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 15:20:57.497589 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494936 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 15:20:57.497589 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494940 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 15:20:57.497589 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494944 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 15:20:57.497589 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.494948 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 15:20:57.497589 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497098 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 15:20:57.497589 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497111 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 15:20:57.497589 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497117 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 15:20:57.498396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497122 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 15:20:57.498396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497127 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 15:20:57.498396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497132 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 15:20:57.498396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497136 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 15:20:57.498396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497140 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 15:20:57.498396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497145 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 15:20:57.498396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497148 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 15:20:57.498396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497153 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 15:20:57.498396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497157 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 15:20:57.498396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497161 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 15:20:57.498396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497165 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 15:20:57.498396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497170 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 15:20:57.498396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497174 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 15:20:57.498396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497178 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 15:20:57.498396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497184 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 15:20:57.498396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497188 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 15:20:57.498396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497193 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 15:20:57.498396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497197 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 15:20:57.498396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497202 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 15:20:57.498396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497207 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 14 15:20:57.499285 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497212 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 15:20:57.499285 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497218 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 15:20:57.499285 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497224 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 15:20:57.499285 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497230 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 15:20:57.499285 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497235 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 15:20:57.499285 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497242 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 15:20:57.499285 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497248 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 15:20:57.499285 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497253 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 15:20:57.499285 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497258 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 15:20:57.499285 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497262 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 15:20:57.499285 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497266 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 15:20:57.499285 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497270 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 15:20:57.499285 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497275 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 15:20:57.499285 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497279 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 15:20:57.499285 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497283 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 15:20:57.499285 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497287 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 15:20:57.499285 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497291 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 15:20:57.499285 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497296 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 15:20:57.499285 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497300 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 15:20:57.500131 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497304 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 15:20:57.500131 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497308 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 15:20:57.500131 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497312 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 15:20:57.500131 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497316 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 15:20:57.500131 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497321 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 15:20:57.500131 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497325 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 15:20:57.500131 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497329 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 15:20:57.500131 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497333 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 15:20:57.500131 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497339 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 15:20:57.500131 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497345 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 15:20:57.500131 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497349 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 15:20:57.500131 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497353 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 15:20:57.500131 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497357 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 15:20:57.500131 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497362 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 15:20:57.500131 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497366 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 15:20:57.500131 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497370 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 15:20:57.500131 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497376 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 15:20:57.500131 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497380 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 15:20:57.500131 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497385 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 15:20:57.500131 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497389 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 15:20:57.500792 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497393 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 15:20:57.500792 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497399 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 15:20:57.500792 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497403 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 15:20:57.500792 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497407 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 15:20:57.500792 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497412 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 15:20:57.500792 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497416 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 15:20:57.500792 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497422 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 15:20:57.500792 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497428 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 15:20:57.500792 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497432 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 15:20:57.500792 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497437 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 15:20:57.500792 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497441 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 15:20:57.500792 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497445 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 15:20:57.500792 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497449 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 15:20:57.500792 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497453 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 15:20:57.500792 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497457 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 15:20:57.500792 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497461 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 15:20:57.500792 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497466 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 15:20:57.500792 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497470 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 15:20:57.500792 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497474 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 15:20:57.500792 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497478 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 15:20:57.501558 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497482 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 15:20:57.501558 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497487 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 15:20:57.501558 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497491 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 15:20:57.501558 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497495 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 15:20:57.501558 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497499 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 15:20:57.501558 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497503 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 15:20:57.501558 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497507 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 15:20:57.501558 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497511 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 15:20:57.501558 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497516 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 15:20:57.501558 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497520 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 15:20:57.501558 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497524 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 15:20:57.501558 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497528 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 15:20:57.501558 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497533 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 15:20:57.501558 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497537 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 15:20:57.501558 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497541 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 15:20:57.501558 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497545 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 15:20:57.501558 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.497549 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 15:20:57.501558 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498196 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 15:20:57.501558 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498206 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 15:20:57.501558 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498211 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 15:20:57.502073 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498215 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 15:20:57.502073 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498219 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 15:20:57.502073 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498224 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 15:20:57.502073 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498228 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 15:20:57.502073 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498233 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 15:20:57.502073 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498237 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 15:20:57.502073 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498241 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 15:20:57.502073 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498245 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 15:20:57.502073 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498249 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 15:20:57.502073 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498254 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 15:20:57.502073 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498258 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 15:20:57.502073 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498262 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 15:20:57.502073 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498267 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 15:20:57.502073 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498272 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 15:20:57.502073 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498276 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 15:20:57.502073 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498281 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 15:20:57.502073 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498285 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 15:20:57.502073 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498289 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 15:20:57.502073 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498293 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 14 15:20:57.502073 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498297 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 15:20:57.502643 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498302 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 15:20:57.502643 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498307 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 15:20:57.502643 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498311 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 15:20:57.502643 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498315 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 15:20:57.502643 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498319 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 15:20:57.502643 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498323 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 15:20:57.502643 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498327 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 15:20:57.502643 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498331 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 15:20:57.502643 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498335 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 15:20:57.502643 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498339 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 15:20:57.502643 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498343 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 15:20:57.502643 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498347 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 15:20:57.502643 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498351 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 15:20:57.502643 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498355 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 15:20:57.502643 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498359 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 15:20:57.502643 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498363 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 15:20:57.502643 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498368 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 15:20:57.502643 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498372 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 15:20:57.502643 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498376 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 15:20:57.502643 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498380 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498384 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498388 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498392 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498395 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498399 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498404 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498409 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498413 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498417 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498421 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498425 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498429 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498433 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498438 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498442 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498447 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498451 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498455 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498459 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498463 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 15:20:57.503184 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498467 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 15:20:57.503703 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498471 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 15:20:57.503703 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498476 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 15:20:57.503703 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498481 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 15:20:57.503703 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498488 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 15:20:57.503703 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498494 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 15:20:57.503703 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498498 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 15:20:57.503703 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498502 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 15:20:57.503703 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498506 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 15:20:57.503703 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498510 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 15:20:57.503703 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498515 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 15:20:57.503703 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498520 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 15:20:57.503703 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498527 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 15:20:57.503703 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498531 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 15:20:57.503703 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498540 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 15:20:57.503703 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498545 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 15:20:57.503703 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498550 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 15:20:57.503703 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498555 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 15:20:57.503703 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498560 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498566 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498570 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498575 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498579 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498583 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498587 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498591 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498595 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498600 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498604 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498608 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498612 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498616 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498621 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498625 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498629 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.498633 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498728 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498748 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498761 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 14 15:20:57.504159 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498768 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498775 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498780 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498786 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498793 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498798 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498803 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498808 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498814 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498818 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498823 2579 flags.go:64] FLAG: --cgroup-root="" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498828 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498832 2579 flags.go:64] FLAG: --client-ca-file="" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498837 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498842 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498847 2579 flags.go:64] FLAG: --cluster-domain="" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498852 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498857 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498862 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498867 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498873 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498877 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498882 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 14 15:20:57.504691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498887 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498891 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498896 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498901 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498906 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498910 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498916 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498921 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498925 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498929 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498934 2579 flags.go:64] FLAG: --enable-server="true" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498938 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498945 2579 flags.go:64] FLAG: --event-burst="100" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498950 2579 flags.go:64] FLAG: --event-qps="50" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498955 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498959 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498964 2579 flags.go:64] FLAG: --eviction-hard="" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498969 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498979 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498984 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498989 2579 flags.go:64] FLAG: --eviction-soft="" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498994 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.498998 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499003 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499008 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 14 15:20:57.505264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499012 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499017 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499021 2579 flags.go:64] FLAG: --feature-gates="" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499047 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499052 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499057 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499061 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499066 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499071 2579 flags.go:64] FLAG: --help="false" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499075 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-128-22.ec2.internal" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499080 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499084 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499089 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499094 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499100 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499104 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499109 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499114 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499118 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499123 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499128 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499133 2579 flags.go:64] FLAG: --kube-reserved="" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499138 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499142 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 14 15:20:57.505945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499147 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499153 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499158 2579 flags.go:64] FLAG: --lock-file="" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499162 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499167 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499171 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499180 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499185 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499190 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499194 2579 flags.go:64] FLAG: --logging-format="text" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499199 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499204 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499209 2579 flags.go:64] FLAG: --manifest-url="" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499213 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499220 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499224 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499231 2579 flags.go:64] FLAG: --max-pods="110" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499236 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499240 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499245 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499249 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499254 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499259 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499263 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499274 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 14 15:20:57.506574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499279 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499283 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499287 2579 flags.go:64] FLAG: --pod-cidr="" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499292 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499299 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499303 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499308 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499312 2579 flags.go:64] FLAG: --port="10250" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499317 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499323 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0538d8064ef0b6c8f" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499328 2579 flags.go:64] FLAG: --qos-reserved="" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499333 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499338 2579 flags.go:64] FLAG: --register-node="true" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499343 2579 flags.go:64] FLAG: --register-with-taints="" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499348 2579 flags.go:64] FLAG: --registry-burst="10" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499353 2579 flags.go:64] FLAG: --registry-qps="5" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499358 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499364 2579 flags.go:64] FLAG: --reserved-memory="" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499371 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499376 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499381 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499385 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499390 2579 flags.go:64] FLAG: --runonce="false" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499394 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499399 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 14 15:20:57.507190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499404 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499408 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499413 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499418 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499423 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499427 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499432 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499436 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499440 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499445 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499450 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499454 2579 flags.go:64] FLAG: --system-cgroups="" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499458 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499467 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499477 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499481 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499489 2579 flags.go:64] FLAG: --tls-min-version="" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499493 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499498 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499503 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499507 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499512 2579 flags.go:64] FLAG: --v="2" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499518 2579 flags.go:64] FLAG: --version="false" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499526 2579 flags.go:64] FLAG: --vmodule="" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499534 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 14 15:20:57.507782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.499540 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 14 15:20:57.508443 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499715 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 15:20:57.508443 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499722 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 15:20:57.508443 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499726 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 15:20:57.508443 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499730 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 15:20:57.508443 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499735 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 15:20:57.508443 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499740 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 15:20:57.508443 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499745 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 15:20:57.508443 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499749 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 15:20:57.508443 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499754 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 15:20:57.508443 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499758 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 15:20:57.508443 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499763 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 15:20:57.508443 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499767 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 15:20:57.508443 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499772 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 15:20:57.508443 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499776 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 15:20:57.508443 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499781 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 15:20:57.508443 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499785 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 15:20:57.508443 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499789 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 15:20:57.508443 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499793 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 15:20:57.508443 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499797 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 15:20:57.508443 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499801 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 15:20:57.508945 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499805 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 15:20:57.508945 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499809 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 15:20:57.508945 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499816 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 15:20:57.508945 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499820 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 15:20:57.508945 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499824 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 15:20:57.508945 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499828 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 15:20:57.508945 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499832 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 15:20:57.508945 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499836 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 15:20:57.508945 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499840 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 15:20:57.508945 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499846 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 15:20:57.508945 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499854 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 15:20:57.508945 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499859 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 15:20:57.508945 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499864 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 15:20:57.508945 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499869 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 15:20:57.508945 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499874 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 15:20:57.508945 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499879 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 15:20:57.508945 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499885 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 15:20:57.508945 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499889 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 15:20:57.508945 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499893 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 15:20:57.509436 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499897 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 15:20:57.509436 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499902 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 15:20:57.509436 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499906 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 15:20:57.509436 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499919 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 15:20:57.509436 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499923 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 15:20:57.509436 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499927 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 15:20:57.509436 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499931 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 15:20:57.509436 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499935 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 15:20:57.509436 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499940 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 15:20:57.509436 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499944 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 15:20:57.509436 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499948 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 15:20:57.509436 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499954 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 15:20:57.509436 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499960 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 15:20:57.509436 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499965 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 15:20:57.509436 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499969 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 15:20:57.509436 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499974 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 15:20:57.509436 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499979 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 15:20:57.509436 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499983 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 15:20:57.509436 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499987 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 15:20:57.509436 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499991 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 15:20:57.509924 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499995 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 15:20:57.509924 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.499999 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 15:20:57.509924 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500004 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 15:20:57.509924 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500010 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 15:20:57.509924 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500015 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 15:20:57.509924 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500020 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 15:20:57.509924 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500040 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 15:20:57.509924 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500055 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 15:20:57.509924 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500059 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 15:20:57.509924 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500063 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 15:20:57.509924 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500068 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 14 15:20:57.509924 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500072 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 15:20:57.509924 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500076 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 15:20:57.509924 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500081 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 15:20:57.509924 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500085 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 15:20:57.509924 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500089 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 15:20:57.509924 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500092 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 15:20:57.509924 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500096 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 15:20:57.509924 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500101 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 15:20:57.509924 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500105 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 15:20:57.510439 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500109 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 15:20:57.510439 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500116 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 15:20:57.510439 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500121 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 15:20:57.510439 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500126 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 15:20:57.510439 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500131 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 15:20:57.510439 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500135 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 15:20:57.510439 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500139 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 15:20:57.510439 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500145 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 15:20:57.510439 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500149 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 15:20:57.510439 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500153 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 15:20:57.510439 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500158 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 15:20:57.510439 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500162 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 15:20:57.510439 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500167 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 15:20:57.510439 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500171 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 15:20:57.510439 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500175 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 15:20:57.510439 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500179 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 15:20:57.510439 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500185 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 15:20:57.510439 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500190 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 15:20:57.510439 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500195 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 15:20:57.510906 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.500199 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 15:20:57.510906 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.500208 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 15:20:57.510906 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.507101 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 14 15:20:57.510906 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.507119 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 14 15:20:57.510906 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.507968 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 14 15:20:57.510906 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.507976 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 14 15:20:57.510906 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508048 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 15:20:57.510906 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508054 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 15:20:57.510906 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508057 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 15:20:57.510906 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508061 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 15:20:57.510906 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508063 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 15:20:57.510906 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508066 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 15:20:57.510906 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508069 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 15:20:57.510906 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508072 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 15:20:57.511277 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508075 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 15:20:57.511277 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508077 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 15:20:57.511277 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508080 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 15:20:57.511277 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508082 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 15:20:57.511277 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508085 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 15:20:57.511277 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508087 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 15:20:57.511277 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508090 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 15:20:57.511277 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508092 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 14 15:20:57.511277 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508094 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 15:20:57.511277 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508097 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 15:20:57.511277 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508100 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 15:20:57.511277 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508102 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 15:20:57.511277 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508104 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 15:20:57.511277 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508107 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 15:20:57.511277 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508109 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 15:20:57.511277 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508112 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 15:20:57.511277 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508114 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 15:20:57.511277 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508117 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 15:20:57.511277 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508119 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 15:20:57.511277 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508123 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 15:20:57.511755 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508125 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 15:20:57.511755 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508130 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 15:20:57.511755 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508132 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 15:20:57.511755 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508135 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 15:20:57.511755 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508137 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 15:20:57.511755 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508139 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 15:20:57.511755 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508142 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 15:20:57.511755 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508144 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 15:20:57.511755 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508147 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 15:20:57.511755 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508149 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 15:20:57.511755 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508152 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 15:20:57.511755 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508154 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 15:20:57.511755 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508156 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 15:20:57.511755 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508159 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 15:20:57.511755 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508161 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 15:20:57.511755 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508164 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 15:20:57.511755 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508166 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 15:20:57.511755 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508168 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 15:20:57.511755 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508171 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 15:20:57.511755 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508173 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 15:20:57.512260 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508176 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 15:20:57.512260 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508178 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 15:20:57.512260 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508181 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 15:20:57.512260 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508183 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 15:20:57.512260 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508186 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 15:20:57.512260 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508188 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 15:20:57.512260 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508191 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 15:20:57.512260 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508193 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 15:20:57.512260 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508197 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 15:20:57.512260 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508201 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 15:20:57.512260 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508204 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 15:20:57.512260 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508206 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 15:20:57.512260 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508209 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 15:20:57.512260 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508212 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 15:20:57.512260 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508214 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 15:20:57.512260 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508217 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 15:20:57.512260 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508219 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 15:20:57.512260 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508222 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 15:20:57.512260 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508225 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 15:20:57.512713 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508229 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 15:20:57.512713 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508232 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 15:20:57.512713 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508235 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 15:20:57.512713 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508237 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 15:20:57.512713 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508239 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 15:20:57.512713 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508242 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 15:20:57.512713 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508244 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 15:20:57.512713 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508247 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 15:20:57.512713 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508249 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 15:20:57.512713 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508252 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 15:20:57.512713 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508254 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 15:20:57.512713 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508256 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 15:20:57.512713 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508259 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 15:20:57.512713 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508262 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 15:20:57.512713 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508265 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 15:20:57.512713 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508268 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 15:20:57.512713 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508271 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 15:20:57.512713 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508274 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 15:20:57.512713 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508277 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 15:20:57.512713 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508279 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 15:20:57.513206 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508281 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 15:20:57.513206 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508284 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 15:20:57.513206 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508286 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 15:20:57.513206 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508288 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 15:20:57.513206 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508291 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 15:20:57.513206 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508294 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 15:20:57.513206 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508296 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 15:20:57.513206 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508299 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 15:20:57.513206 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508302 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 15:20:57.513206 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508304 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 15:20:57.513206 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508307 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 15:20:57.513206 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508309 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 15:20:57.513206 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.508314 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 15:20:57.513206 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508460 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 15:20:57.513206 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508466 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 15:20:57.513571 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508469 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 15:20:57.513571 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508471 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 15:20:57.513571 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508474 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 15:20:57.513571 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508477 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 15:20:57.513571 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508479 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 15:20:57.513571 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508482 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 15:20:57.513571 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508485 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 15:20:57.513571 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508487 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 15:20:57.513571 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508489 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 15:20:57.513571 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508492 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 15:20:57.513571 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508494 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 15:20:57.513571 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508497 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 15:20:57.513571 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508499 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 15:20:57.513571 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508502 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 15:20:57.513571 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508504 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 15:20:57.513571 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508506 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 15:20:57.513571 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508509 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 15:20:57.513571 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508511 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 15:20:57.513571 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508514 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 15:20:57.513571 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508516 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 15:20:57.514060 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508519 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 15:20:57.514060 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508521 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 15:20:57.514060 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508524 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 15:20:57.514060 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508526 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 15:20:57.514060 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508529 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 15:20:57.514060 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508531 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 15:20:57.514060 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508540 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 15:20:57.514060 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508542 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 15:20:57.514060 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508545 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 15:20:57.514060 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508548 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 15:20:57.514060 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508550 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 15:20:57.514060 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508553 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 15:20:57.514060 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508555 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 15:20:57.514060 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508558 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 15:20:57.514060 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508560 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 15:20:57.514060 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508562 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 15:20:57.514060 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508564 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 15:20:57.514060 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508567 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 15:20:57.514060 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508569 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 15:20:57.514060 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508572 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 15:20:57.514538 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508574 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 15:20:57.514538 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508578 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 15:20:57.514538 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508581 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 15:20:57.514538 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508584 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 15:20:57.514538 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508586 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 15:20:57.514538 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508589 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 15:20:57.514538 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508591 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 15:20:57.514538 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508594 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 15:20:57.514538 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508596 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 15:20:57.514538 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508598 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 15:20:57.514538 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508601 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 15:20:57.514538 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508603 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 15:20:57.514538 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508607 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 15:20:57.514538 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508610 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 15:20:57.514538 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508613 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 15:20:57.514538 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508615 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 15:20:57.514538 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508618 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 15:20:57.514538 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508620 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 15:20:57.514538 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508623 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 15:20:57.514992 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508647 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 15:20:57.514992 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508651 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 15:20:57.514992 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508654 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 15:20:57.514992 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508657 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 15:20:57.514992 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508659 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 15:20:57.514992 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508662 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 15:20:57.514992 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508665 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 15:20:57.514992 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508668 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 15:20:57.514992 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508670 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 15:20:57.514992 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508673 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 15:20:57.514992 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508676 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 15:20:57.514992 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508678 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 15:20:57.514992 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508680 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 14 15:20:57.514992 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508683 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 15:20:57.514992 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508685 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 15:20:57.514992 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508689 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 15:20:57.514992 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508692 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 15:20:57.514992 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508695 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 15:20:57.514992 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508698 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 15:20:57.514992 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508700 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 15:20:57.515526 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508702 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 15:20:57.515526 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508705 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 15:20:57.515526 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508707 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 15:20:57.515526 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508709 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 15:20:57.515526 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508712 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 15:20:57.515526 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508714 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 15:20:57.515526 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508716 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 15:20:57.515526 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508719 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 15:20:57.515526 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508721 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 15:20:57.515526 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508724 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 15:20:57.515526 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508727 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 15:20:57.515526 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508729 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 15:20:57.515526 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508731 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 15:20:57.515526 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508740 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 15:20:57.515526 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508742 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 15:20:57.515526 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508744 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 15:20:57.515526 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508747 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 15:20:57.515526 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:57.508749 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 15:20:57.515967 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.508754 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 15:20:57.515967 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.508877 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 14 15:20:57.515967 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.510909 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 14 15:20:57.515967 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.512310 2579 server.go:1019] "Starting client certificate rotation" Jul 14 15:20:57.515967 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.512436 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 15:20:57.515967 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.512474 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 15:20:57.515967 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.515419 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 15:20:57.521086 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.521059 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 15:20:57.534702 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.534683 2579 log.go:25] "Validated CRI v1 runtime API" Jul 14 15:20:57.537064 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.537043 2579 log.go:25] "Validated CRI v1 image API" Jul 14 15:20:57.538705 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.538681 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 14 15:20:57.541210 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.541190 2579 fs.go:135] Filesystem UUIDs: map[07048312-ed03-47a8-8452-466bf3048ec5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b28a5477-9b98-49de-82da-8ddaab491d35:/dev/nvme0n1p3] Jul 14 15:20:57.541298 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.541208 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 14 15:20:57.544484 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.544467 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 15:20:57.546619 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.546511 2579 manager.go:216] Machine: {Timestamp:2026-07-14 15:20:57.545070247 +0000 UTC m=+0.379597387 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110958 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec280846aa99f0f64467bf815a944dda SystemUUID:ec280846-aa99-f0f6-4467-bf815a944dda BootID:28a2d304-3bc8-45cd-bb38-659a8f4ce460 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:62:40:bd:bf:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:62:40:bd:bf:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:57:30:54:85:f2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 14 15:20:57.546619 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.546617 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 14 15:20:57.546722 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.546692 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 14 15:20:57.549993 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.549968 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 14 15:20:57.550157 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.549996 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-22.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 14 15:20:57.550791 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.550782 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 14 15:20:57.550823 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.550795 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 14 15:20:57.550823 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.550802 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 15:20:57.551706 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.551696 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 15:20:57.553266 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.553255 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 14 15:20:57.556566 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.556554 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 14 15:20:57.556691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.556681 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 14 15:20:57.556727 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.556720 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 14 15:20:57.556754 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.556734 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 14 15:20:57.556754 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.556748 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 14 15:20:57.556822 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.556757 2579 kubelet.go:398] "Adding apiserver pod source" Jul 14 15:20:57.556822 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.556767 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 14 15:20:57.557209 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.557194 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 15:20:57.557287 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.557218 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 15:20:57.560243 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.560225 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 14 15:20:57.562237 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.562219 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 14 15:20:57.562325 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.562252 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 14 15:20:57.562425 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.562414 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 14 15:20:57.562459 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.562431 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 14 15:20:57.562459 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.562437 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 14 15:20:57.562459 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.562442 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 14 15:20:57.562459 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.562450 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 14 15:20:57.562459 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.562456 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 14 15:20:57.562982 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.562973 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 14 15:20:57.563010 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.562983 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 14 15:20:57.563010 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.562989 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 14 15:20:57.563010 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.562995 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 14 15:20:57.563010 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.563003 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 14 15:20:57.563010 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.563010 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 14 15:20:57.563861 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.563849 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 14 15:20:57.563898 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.563864 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 14 15:20:57.567156 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.567142 2579 server.go:1268] "Started kubelet" Jul 14 15:20:57.567254 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.567214 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 14 15:20:57.567318 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.567211 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 14 15:20:57.567318 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.567285 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 14 15:20:57.568005 ip-10-0-128-22 systemd[1]: Started Kubernetes Kubelet. Jul 14 15:20:57.574206 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:57.574175 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 14 15:20:57.574395 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.574379 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-22.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 14 15:20:57.574728 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:57.574703 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-22.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 14 15:20:57.574854 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.574769 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 14 15:20:57.575331 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:57.575308 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 14 15:20:57.575820 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.575794 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 14 15:20:57.579001 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:57.577993 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-22.ec2.internal.18c23129b380c199 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-22.ec2.internal,UID:ip-10-0-128-22.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-22.ec2.internal,},FirstTimestamp:2026-07-14 15:20:57.567117721 +0000 UTC m=+0.401644860,LastTimestamp:2026-07-14 15:20:57.567117721 +0000 UTC m=+0.401644860,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-22.ec2.internal,}" Jul 14 15:20:57.580758 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.580741 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 14 15:20:57.580836 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.580752 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 14 15:20:57.581169 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:57.580960 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 14 15:20:57.581482 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.581462 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 14 15:20:57.581482 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.581482 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 14 15:20:57.581633 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.581607 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 14 15:20:57.581687 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.581659 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 14 15:20:57.581687 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.581669 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 14 15:20:57.581865 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.581847 2579 factory.go:153] Registering CRI-O factory Jul 14 15:20:57.581865 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.581867 2579 factory.go:223] Registration of the crio container factory successfully Jul 14 15:20:57.581980 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.581908 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 14 15:20:57.581980 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.581915 2579 factory.go:55] Registering systemd factory Jul 14 15:20:57.581980 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.581921 2579 factory.go:223] Registration of the systemd container factory successfully Jul 14 15:20:57.581980 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.581937 2579 factory.go:103] Registering Raw factory Jul 14 15:20:57.581980 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.581946 2579 manager.go:1196] Started watching for new ooms in manager Jul 14 15:20:57.582478 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.582464 2579 manager.go:349] Starting recovery of all containers Jul 14 15:20:57.591286 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.591160 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zptmv" Jul 14 15:20:57.592574 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:57.592533 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-22.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 14 15:20:57.592680 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:57.592599 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 14 15:20:57.593409 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.593393 2579 manager.go:354] Recovery completed Jul 14 15:20:57.597127 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.597112 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zptmv" Jul 14 15:20:57.597848 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.597837 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 15:20:57.599512 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.599495 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientMemory" Jul 14 15:20:57.599587 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.599529 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 15:20:57.599587 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.599544 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientPID" Jul 14 15:20:57.600069 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.600054 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 14 15:20:57.600140 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.600071 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 14 15:20:57.600140 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.600059 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 15:20:57.600140 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.600086 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 14 15:20:57.600767 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.600753 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientMemory" Jul 14 15:20:57.600824 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.600778 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 15:20:57.600824 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.600787 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientPID" Jul 14 15:20:57.601170 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.601158 2579 policy_none.go:49] "None policy: Start" Jul 14 15:20:57.601203 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.601177 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 14 15:20:57.601203 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.601191 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 14 15:20:57.602036 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.602013 2579 policy_none.go:47] "Start" Jul 14 15:20:57.639964 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.639821 2579 manager.go:337] "Starting Device Plugin manager" Jul 14 15:20:57.654785 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:57.639989 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 14 15:20:57.654785 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.640004 2579 server.go:87] "Starting device plugin registration server" Jul 14 15:20:57.654785 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.640242 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 14 15:20:57.654785 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.640255 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 14 15:20:57.654785 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.640364 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 14 15:20:57.654785 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.640451 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 14 15:20:57.654785 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.640460 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 14 15:20:57.654785 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:57.640882 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 14 15:20:57.654785 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:57.640932 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-22.ec2.internal\" not found" Jul 14 15:20:57.706835 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.706797 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 14 15:20:57.708150 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.708134 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 14 15:20:57.708244 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.708175 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 14 15:20:57.708244 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.708206 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 14 15:20:57.708339 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:57.708243 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 14 15:20:57.709864 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.709844 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 15:20:57.741168 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.741121 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 15:20:57.742200 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.742185 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientMemory" Jul 14 15:20:57.742259 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.742215 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 15:20:57.742259 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.742230 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientPID" Jul 14 15:20:57.742259 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.742250 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-22.ec2.internal" Jul 14 15:20:57.753286 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.753268 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-22.ec2.internal" Jul 14 15:20:57.753331 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:57.753288 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-22.ec2.internal\": node \"ip-10-0-128-22.ec2.internal\" not found" Jul 14 15:20:57.767387 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:57.767367 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 14 15:20:57.808914 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.808892 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal"] Jul 14 15:20:57.808974 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.808949 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 15:20:57.810401 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.810387 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientMemory" Jul 14 15:20:57.810448 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.810415 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 15:20:57.810448 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.810425 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientPID" Jul 14 15:20:57.811561 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.811549 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 15:20:57.811675 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.811661 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" Jul 14 15:20:57.811710 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.811691 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 15:20:57.812289 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.812273 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientMemory" Jul 14 15:20:57.812352 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.812302 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 15:20:57.812352 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.812312 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientPID" Jul 14 15:20:57.813360 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.813342 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientMemory" Jul 14 15:20:57.813430 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.813380 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 15:20:57.813430 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.813396 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientPID" Jul 14 15:20:57.814716 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.814701 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal" Jul 14 15:20:57.814775 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.814732 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 15:20:57.818203 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.818189 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientMemory" Jul 14 15:20:57.818282 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.818213 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 15:20:57.818282 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.818227 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientPID" Jul 14 15:20:57.829608 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:57.829588 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-22.ec2.internal\" not found" node="ip-10-0-128-22.ec2.internal" Jul 14 15:20:57.835384 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:57.835370 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-22.ec2.internal\" not found" node="ip-10-0-128-22.ec2.internal" Jul 14 15:20:57.867964 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:57.867945 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 14 15:20:57.883386 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.883368 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83a908925c0c8a87df4dd7fc6977e74c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal\" (UID: \"83a908925c0c8a87df4dd7fc6977e74c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" Jul 14 15:20:57.883457 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.883392 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83a908925c0c8a87df4dd7fc6977e74c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal\" (UID: \"83a908925c0c8a87df4dd7fc6977e74c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" Jul 14 15:20:57.883457 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.883409 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebf13fbb58bd809e707ce060a950b3f9-config\") pod \"kube-apiserver-proxy-ip-10-0-128-22.ec2.internal\" (UID: \"ebf13fbb58bd809e707ce060a950b3f9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal" Jul 14 15:20:57.968279 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:57.968228 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 14 15:20:57.983583 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.983552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83a908925c0c8a87df4dd7fc6977e74c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal\" (UID: \"83a908925c0c8a87df4dd7fc6977e74c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" Jul 14 15:20:57.983668 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.983587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83a908925c0c8a87df4dd7fc6977e74c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal\" (UID: \"83a908925c0c8a87df4dd7fc6977e74c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" Jul 14 15:20:57.983668 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.983606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebf13fbb58bd809e707ce060a950b3f9-config\") pod \"kube-apiserver-proxy-ip-10-0-128-22.ec2.internal\" (UID: \"ebf13fbb58bd809e707ce060a950b3f9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal" Jul 14 15:20:57.983668 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.983654 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83a908925c0c8a87df4dd7fc6977e74c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal\" (UID: \"83a908925c0c8a87df4dd7fc6977e74c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" Jul 14 15:20:57.983756 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.983662 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83a908925c0c8a87df4dd7fc6977e74c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal\" (UID: \"83a908925c0c8a87df4dd7fc6977e74c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" Jul 14 15:20:57.983756 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:57.983718 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebf13fbb58bd809e707ce060a950b3f9-config\") pod \"kube-apiserver-proxy-ip-10-0-128-22.ec2.internal\" (UID: \"ebf13fbb58bd809e707ce060a950b3f9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal" Jul 14 15:20:58.069086 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:58.068987 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 14 15:20:58.131553 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.131521 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" Jul 14 15:20:58.138337 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.138316 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal" Jul 14 15:20:58.169496 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:58.169462 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 14 15:20:58.270052 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:58.270002 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 14 15:20:58.370611 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:58.370547 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 14 15:20:58.471161 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:58.471130 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 14 15:20:58.511650 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.511626 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 14 15:20:58.512204 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.511802 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 15:20:58.551146 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.551125 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 15:20:58.557536 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.557516 2579 apiserver.go:52] "Watching apiserver" Jul 14 15:20:58.569958 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.569933 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 14 15:20:58.572125 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.572094 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-km9wq","openshift-multus/network-metrics-daemon-kg2pm","openshift-network-diagnostics/network-check-target-5lbpx","openshift-network-operator/iptables-alerter-prbkb","kube-system/konnectivity-agent-89g6j","openshift-cluster-node-tuning-operator/tuned-8gcgt","openshift-multus/multus-84dvv","openshift-multus/multus-additional-cni-plugins-kvkjc","openshift-ovn-kubernetes/ovnkube-node-g9j4v","kube-system/global-pull-secret-syncer-785zz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s","openshift-dns/node-resolver-jj7cz"] Jul 14 15:20:58.573633 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.573618 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-km9wq" Jul 14 15:20:58.574885 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.574867 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:20:58.574981 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:58.574939 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kg2pm" podUID="edc388f1-89b5-45bf-9672-3e1a135fed4e" Jul 14 15:20:58.576092 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.576070 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:20:58.576202 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:58.576158 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5lbpx" podUID="edffdada-2745-451b-bf87-3ff5facb7b64" Jul 14 15:20:58.576422 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.576405 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 14 15:20:58.576477 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.576429 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 14 15:20:58.576608 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.576594 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 14 15:20:58.576941 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.576928 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pqq4s\"" Jul 14 15:20:58.577702 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.577687 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-89g6j" Jul 14 15:20:58.580731 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.580667 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ncmmm\"" Jul 14 15:20:58.580850 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.580788 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.580964 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.580950 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 14 15:20:58.581051 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.580979 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" Jul 14 15:20:58.581166 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.580839 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 14 15:20:58.581215 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.581191 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 14 15:20:58.582108 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.582091 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.583114 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.583098 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.583638 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.583619 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 14 15:20:58.583722 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.583649 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 14 15:20:58.584006 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.583993 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jm4ms\"" Jul 14 15:20:58.584417 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.584401 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-prbkb" Jul 14 15:20:58.584784 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.584769 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 14 15:20:58.585085 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.585069 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 14 15:20:58.585162 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.585150 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 14 15:20:58.585215 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.585203 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 14 15:20:58.585306 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.585291 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4qnlv\"" Jul 14 15:20:58.585850 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.585828 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8q5bq\"" Jul 14 15:20:58.585850 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.585845 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 14 15:20:58.585990 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.585861 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 14 15:20:58.585990 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.585984 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.586618 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.586601 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-sysconfig\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.586701 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.586627 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8339b58-8a56-4e52-8722-52249ad123d3-cni-binary-copy\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.586701 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.586651 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-multus-socket-dir-parent\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.586701 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.586686 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8339b58-8a56-4e52-8722-52249ad123d3-multus-daemon-config\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.586814 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.586736 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f3999dd-ca70-4115-8778-3c73e9f14124-host\") pod \"node-ca-km9wq\" (UID: \"3f3999dd-ca70-4115-8778-3c73e9f14124\") " pod="openshift-image-registry/node-ca-km9wq" Jul 14 15:20:58.586814 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.586760 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-sysctl-d\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.586814 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.586777 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8trr\" (UniqueName: \"kubernetes.io/projected/9db566a2-6ba1-4a33-8ce8-3c704b556841-kube-api-access-d8trr\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.586814 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.586791 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36b98868-c196-465a-8561-d10895ae0f3c-agent-certs\") pod \"konnectivity-agent-89g6j\" (UID: \"36b98868-c196-465a-8561-d10895ae0f3c\") " pod="kube-system/konnectivity-agent-89g6j" Jul 14 15:20:58.586814 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.586806 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-kubernetes\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.587091 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.586832 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-host-run-netns\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.587091 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.586865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-multus-conf-dir\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.587091 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.586890 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-etc-kubernetes\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.587091 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.586914 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zv28\" (UniqueName: \"kubernetes.io/projected/d8339b58-8a56-4e52-8722-52249ad123d3-kube-api-access-8zv28\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.587091 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.586938 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f3999dd-ca70-4115-8778-3c73e9f14124-serviceca\") pod \"node-ca-km9wq\" (UID: \"3f3999dd-ca70-4115-8778-3c73e9f14124\") " pod="openshift-image-registry/node-ca-km9wq" Jul 14 15:20:58.587091 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.586963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ttql\" (UniqueName: \"kubernetes.io/projected/3f3999dd-ca70-4115-8778-3c73e9f14124-kube-api-access-7ttql\") pod \"node-ca-km9wq\" (UID: \"3f3999dd-ca70-4115-8778-3c73e9f14124\") " pod="openshift-image-registry/node-ca-km9wq" Jul 14 15:20:58.587091 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.586986 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs\") pod \"network-metrics-daemon-kg2pm\" (UID: \"edc388f1-89b5-45bf-9672-3e1a135fed4e\") " pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:20:58.587091 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-sysctl-conf\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.587091 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587048 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-tuned\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.587492 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587091 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-host-run-k8s-cni-cncf-io\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.587492 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587153 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-host-var-lib-cni-multus\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.587492 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587168 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36b98868-c196-465a-8561-d10895ae0f3c-konnectivity-ca\") pod \"konnectivity-agent-89g6j\" (UID: \"36b98868-c196-465a-8561-d10895ae0f3c\") " pod="kube-system/konnectivity-agent-89g6j" Jul 14 15:20:58.587492 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587198 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-lib-modules\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.587492 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587226 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-var-lib-kubelet\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.587492 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587252 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9db566a2-6ba1-4a33-8ce8-3c704b556841-tmp\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.587492 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587262 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 14 15:20:58.587492 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587274 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-cnibin\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.587492 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587295 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-modprobe-d\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.587492 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587327 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-systemd\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.587492 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587355 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-host\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.587492 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587371 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:20:58.587492 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587386 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-os-release\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.587492 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587408 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 14 15:20:58.587492 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587416 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-host-var-lib-cni-bin\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.587492 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587422 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xlsmx\"" Jul 14 15:20:58.587492 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:58.587433 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785zz" podUID="f42d7bd8-f742-4b0a-83d9-2e0f121bda84" Jul 14 15:20:58.587492 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45g9f\" (UniqueName: \"kubernetes.io/projected/edc388f1-89b5-45bf-9672-3e1a135fed4e-kube-api-access-45g9f\") pod \"network-metrics-daemon-kg2pm\" (UID: \"edc388f1-89b5-45bf-9672-3e1a135fed4e\") " pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:20:58.587492 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587470 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 14 15:20:58.588148 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587478 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-run\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.588148 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587511 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-multus-cni-dir\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.588148 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587544 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-host-var-lib-kubelet\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.588148 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587573 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-hostroot\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.588148 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-sys\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.588148 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587618 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-system-cni-dir\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.588148 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587659 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-host-run-multus-certs\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.588148 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.587685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gw7t\" (UniqueName: \"kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t\") pod \"network-check-target-5lbpx\" (UID: \"edffdada-2745-451b-bf87-3ff5facb7b64\") " pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:20:58.588514 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.588499 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 14 15:20:58.588610 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.588592 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 14 15:20:58.588872 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.588853 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 14 15:20:58.588972 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.588891 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 14 15:20:58.588972 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.588927 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 14 15:20:58.588972 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.588938 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 14 15:20:58.589522 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.589233 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nf545\"" Jul 14 15:20:58.590513 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.590491 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.591855 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.591671 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jj7cz" Jul 14 15:20:58.593780 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.593392 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 14 15:20:58.593780 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.593574 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 14 15:20:58.593780 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.593773 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mplrs\"" Jul 14 15:20:58.595020 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.594398 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 14 15:20:58.595020 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.594411 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 14 15:20:58.595020 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.594555 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9jmmp\"" Jul 14 15:20:58.595020 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.594609 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 14 15:20:58.595391 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.595179 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 15:20:58.598549 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.598531 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 15:20:58.599753 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.599736 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 15:20:58.599835 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.599807 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal" Jul 14 15:20:58.600130 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.600113 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal"] Jul 14 15:20:58.600173 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.600120 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 15:15:59 +0000 UTC" deadline="2028-04-13 11:40:36.360066972 +0000 UTC" Jul 14 15:20:58.600173 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.600158 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15332h19m37.759911588s" Jul 14 15:20:58.607263 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.607243 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal"] Jul 14 15:20:58.607538 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.607525 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 15:20:58.620275 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.620256 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dhn6r" Jul 14 15:20:58.628685 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.628629 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dhn6r" Jul 14 15:20:58.682053 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:58.682008 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf13fbb58bd809e707ce060a950b3f9.slice/crio-4c9cbd241d0b2962853f254746fdb818239014cf499473c544154ce66109cbcb WatchSource:0}: Error finding container 4c9cbd241d0b2962853f254746fdb818239014cf499473c544154ce66109cbcb: Status 404 returned error can't find the container with id 4c9cbd241d0b2962853f254746fdb818239014cf499473c544154ce66109cbcb Jul 14 15:20:58.682253 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.682239 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 14 15:20:58.687846 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.687822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-kubernetes\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.687943 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.687856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-host-run-netns\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.687943 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.687875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-multus-conf-dir\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.687943 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.687889 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-etc-kubernetes\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.687943 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.687903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zv28\" (UniqueName: \"kubernetes.io/projected/d8339b58-8a56-4e52-8722-52249ad123d3-kube-api-access-8zv28\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.687943 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.687900 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-kubernetes\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.687943 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.687930 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ttql\" (UniqueName: \"kubernetes.io/projected/3f3999dd-ca70-4115-8778-3c73e9f14124-kube-api-access-7ttql\") pod \"node-ca-km9wq\" (UID: \"3f3999dd-ca70-4115-8778-3c73e9f14124\") " pod="openshift-image-registry/node-ca-km9wq" Jul 14 15:20:58.687943 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.687931 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-host-run-netns\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.687943 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.687947 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36b98868-c196-465a-8561-d10895ae0f3c-konnectivity-ca\") pod \"konnectivity-agent-89g6j\" (UID: \"36b98868-c196-465a-8561-d10895ae0f3c\") " pod="kube-system/konnectivity-agent-89g6j" Jul 14 15:20:58.688398 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.687967 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-etc-kubernetes\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.688398 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.687981 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdtzf\" (UniqueName: \"kubernetes.io/projected/261d33cd-4df1-4f4a-92b0-fcdeece8956b-kube-api-access-zdtzf\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.688398 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.687993 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-multus-conf-dir\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.688398 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-sysctl-conf\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.688398 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688057 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae601732-7bc1-4c7b-8901-7ff3feebb722-iptables-alerter-script\") pod \"iptables-alerter-prbkb\" (UID: \"ae601732-7bc1-4c7b-8901-7ff3feebb722\") " pod="openshift-network-operator/iptables-alerter-prbkb" Jul 14 15:20:58.688398 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688080 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-systemd-units\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.688398 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688104 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbkrl\" (UniqueName: \"kubernetes.io/projected/db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb-kube-api-access-lbkrl\") pod \"node-resolver-jj7cz\" (UID: \"db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb\") " pod="openshift-dns/node-resolver-jj7cz" Jul 14 15:20:58.688398 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688126 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-etc-openvswitch\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.688398 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688150 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret\") pod \"global-pull-secret-syncer-785zz\" (UID: \"f42d7bd8-f742-4b0a-83d9-2e0f121bda84\") " pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:20:58.688398 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688171 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5062eb6e-d839-4164-aad5-057e03ade017-system-cni-dir\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.688398 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688183 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-sysctl-conf\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.688398 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688193 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5062eb6e-d839-4164-aad5-057e03ade017-cnibin\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.688398 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688217 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5062eb6e-d839-4164-aad5-057e03ade017-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.688398 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-host\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.688398 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688282 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45g9f\" (UniqueName: \"kubernetes.io/projected/edc388f1-89b5-45bf-9672-3e1a135fed4e-kube-api-access-45g9f\") pod \"network-metrics-daemon-kg2pm\" (UID: \"edc388f1-89b5-45bf-9672-3e1a135fed4e\") " pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:20:58.688398 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688296 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-host\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.688398 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688308 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-kubelet\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.689304 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688330 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-run\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.689304 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-multus-cni-dir\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.689304 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688372 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-host-var-lib-kubelet\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.689304 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-hostroot\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.689304 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688416 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxdtp\" (UniqueName: \"kubernetes.io/projected/ae601732-7bc1-4c7b-8901-7ff3feebb722-kube-api-access-sxdtp\") pod \"iptables-alerter-prbkb\" (UID: \"ae601732-7bc1-4c7b-8901-7ff3feebb722\") " pod="openshift-network-operator/iptables-alerter-prbkb" Jul 14 15:20:58.689304 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688439 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.689304 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688460 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c16af064-1d6b-4819-b9dd-92087eab85c0-sys-fs\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.689304 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688464 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36b98868-c196-465a-8561-d10895ae0f3c-konnectivity-ca\") pod \"konnectivity-agent-89g6j\" (UID: \"36b98868-c196-465a-8561-d10895ae0f3c\") " pod="kube-system/konnectivity-agent-89g6j" Jul 14 15:20:58.689304 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688479 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48mqx\" (UniqueName: \"kubernetes.io/projected/c16af064-1d6b-4819-b9dd-92087eab85c0-kube-api-access-48mqx\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.689304 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688502 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-host-run-multus-certs\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.689304 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688522 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-multus-cni-dir\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.689304 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688524 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-dbus\") pod \"global-pull-secret-syncer-785zz\" (UID: \"f42d7bd8-f742-4b0a-83d9-2e0f121bda84\") " pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:20:58.689304 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688569 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-host-var-lib-kubelet\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.689304 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688588 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-hostroot\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.689304 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688602 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-host-run-multus-certs\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.689304 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688607 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5062eb6e-d839-4164-aad5-057e03ade017-os-release\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.689304 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688628 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-run\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.689304 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688634 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 15:20:58.690144 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688640 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c16af064-1d6b-4819-b9dd-92087eab85c0-registration-dir\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.690144 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688665 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-sysconfig\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.690144 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-multus-socket-dir-parent\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.690144 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688735 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae601732-7bc1-4c7b-8901-7ff3feebb722-host-slash\") pod \"iptables-alerter-prbkb\" (UID: \"ae601732-7bc1-4c7b-8901-7ff3feebb722\") " pod="openshift-network-operator/iptables-alerter-prbkb" Jul 14 15:20:58.690144 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688736 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-multus-socket-dir-parent\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.690144 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688748 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-sysconfig\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.690144 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c16af064-1d6b-4819-b9dd-92087eab85c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.690144 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688791 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c16af064-1d6b-4819-b9dd-92087eab85c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.690144 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-sysctl-d\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.690144 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8trr\" (UniqueName: \"kubernetes.io/projected/9db566a2-6ba1-4a33-8ce8-3c704b556841-kube-api-access-d8trr\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.690144 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36b98868-c196-465a-8561-d10895ae0f3c-agent-certs\") pod \"konnectivity-agent-89g6j\" (UID: \"36b98868-c196-465a-8561-d10895ae0f3c\") " pod="kube-system/konnectivity-agent-89g6j" Jul 14 15:20:58.690144 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688923 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-sysctl-d\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.690144 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.688978 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.690144 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-cni-netd\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.690144 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-kubelet-config\") pod \"global-pull-secret-syncer-785zz\" (UID: \"f42d7bd8-f742-4b0a-83d9-2e0f121bda84\") " pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:20:58.690144 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689075 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5062eb6e-d839-4164-aad5-057e03ade017-cni-binary-copy\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.690144 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f3999dd-ca70-4115-8778-3c73e9f14124-serviceca\") pod \"node-ca-km9wq\" (UID: \"3f3999dd-ca70-4115-8778-3c73e9f14124\") " pod="openshift-image-registry/node-ca-km9wq" Jul 14 15:20:58.691199 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689136 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs\") pod \"network-metrics-daemon-kg2pm\" (UID: \"edc388f1-89b5-45bf-9672-3e1a135fed4e\") " pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:20:58.691199 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689132 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 14 15:20:58.691199 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689191 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c16af064-1d6b-4819-b9dd-92087eab85c0-device-dir\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.691199 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-tuned\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.691199 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:58.689242 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 15:20:58.691199 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689288 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-host-run-k8s-cni-cncf-io\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.691199 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-host-run-k8s-cni-cncf-io\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.691199 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:58.689296 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs podName:edc388f1-89b5-45bf-9672-3e1a135fed4e nodeName:}" failed. No retries permitted until 2026-07-14 15:20:59.189276188 +0000 UTC m=+2.023803329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs") pod "network-metrics-daemon-kg2pm" (UID: "edc388f1-89b5-45bf-9672-3e1a135fed4e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 15:20:58.691199 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689354 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-host-var-lib-cni-multus\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.691199 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-run-netns\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.691199 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689456 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-node-log\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.691199 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689479 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-cni-bin\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.691199 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689477 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-host-var-lib-cni-multus\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.691199 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689504 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb-hosts-file\") pod \"node-resolver-jj7cz\" (UID: \"db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb\") " pod="openshift-dns/node-resolver-jj7cz" Jul 14 15:20:58.691199 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689539 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-lib-modules\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.691199 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-var-lib-kubelet\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.691199 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9db566a2-6ba1-4a33-8ce8-3c704b556841-tmp\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.691985 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689584 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f3999dd-ca70-4115-8778-3c73e9f14124-serviceca\") pod \"node-ca-km9wq\" (UID: \"3f3999dd-ca70-4115-8778-3c73e9f14124\") " pod="openshift-image-registry/node-ca-km9wq" Jul 14 15:20:58.691985 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689633 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-var-lib-kubelet\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.691985 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689671 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-lib-modules\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.691985 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-cnibin\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.691985 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689710 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-cnibin\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.691985 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689711 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-run-ovn\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.691985 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689740 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/261d33cd-4df1-4f4a-92b0-fcdeece8956b-ovnkube-config\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.691985 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/261d33cd-4df1-4f4a-92b0-fcdeece8956b-env-overrides\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.691985 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689791 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5062eb6e-d839-4164-aad5-057e03ade017-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.691985 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-modprobe-d\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.691985 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689867 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-systemd\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.691985 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689893 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-os-release\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.691985 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689916 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-host-var-lib-cni-bin\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.691985 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689942 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb-tmp-dir\") pod \"node-resolver-jj7cz\" (UID: \"db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb\") " pod="openshift-dns/node-resolver-jj7cz" Jul 14 15:20:58.691985 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689977 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-systemd\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.691985 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.689985 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfntj\" (UniqueName: \"kubernetes.io/projected/5062eb6e-d839-4164-aad5-057e03ade017-kube-api-access-zfntj\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.691985 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690012 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-slash\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.692903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690045 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-host-var-lib-cni-bin\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.692903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690050 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-modprobe-d\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.692903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690076 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-var-lib-openvswitch\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.692903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690101 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-os-release\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.692903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690118 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/261d33cd-4df1-4f4a-92b0-fcdeece8956b-ovn-node-metrics-cert\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.692903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690142 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c16af064-1d6b-4819-b9dd-92087eab85c0-socket-dir\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.692903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690168 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-sys\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.692903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690194 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-system-cni-dir\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.692903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690221 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9db566a2-6ba1-4a33-8ce8-3c704b556841-sys\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.692903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690223 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gw7t\" (UniqueName: \"kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t\") pod \"network-check-target-5lbpx\" (UID: \"edffdada-2745-451b-bf87-3ff5facb7b64\") " pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:20:58.692903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690268 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8339b58-8a56-4e52-8722-52249ad123d3-system-cni-dir\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.692903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690292 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-run-openvswitch\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.692903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690437 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/261d33cd-4df1-4f4a-92b0-fcdeece8956b-ovnkube-script-lib\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.692903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690472 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8339b58-8a56-4e52-8722-52249ad123d3-cni-binary-copy\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.692903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8339b58-8a56-4e52-8722-52249ad123d3-multus-daemon-config\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.692903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690524 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f3999dd-ca70-4115-8778-3c73e9f14124-host\") pod \"node-ca-km9wq\" (UID: \"3f3999dd-ca70-4115-8778-3c73e9f14124\") " pod="openshift-image-registry/node-ca-km9wq" Jul 14 15:20:58.692903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690563 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-run-systemd\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.693445 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690584 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-log-socket\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.693445 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690606 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5062eb6e-d839-4164-aad5-057e03ade017-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.693445 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690762 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f3999dd-ca70-4115-8778-3c73e9f14124-host\") pod \"node-ca-km9wq\" (UID: \"3f3999dd-ca70-4115-8778-3c73e9f14124\") " pod="openshift-image-registry/node-ca-km9wq" Jul 14 15:20:58.693445 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.690994 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8339b58-8a56-4e52-8722-52249ad123d3-cni-binary-copy\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.693445 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.691253 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8339b58-8a56-4e52-8722-52249ad123d3-multus-daemon-config\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.693445 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.692959 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9db566a2-6ba1-4a33-8ce8-3c704b556841-etc-tuned\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.693445 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.692991 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9db566a2-6ba1-4a33-8ce8-3c704b556841-tmp\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.693445 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.693147 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36b98868-c196-465a-8561-d10895ae0f3c-agent-certs\") pod \"konnectivity-agent-89g6j\" (UID: \"36b98868-c196-465a-8561-d10895ae0f3c\") " pod="kube-system/konnectivity-agent-89g6j" Jul 14 15:20:58.705748 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:58.705719 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 15:20:58.705890 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:58.705868 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 15:20:58.705890 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:58.705892 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8gw7t for pod openshift-network-diagnostics/network-check-target-5lbpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 15:20:58.706073 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:58.705966 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t podName:edffdada-2745-451b-bf87-3ff5facb7b64 nodeName:}" failed. No retries permitted until 2026-07-14 15:20:59.205948779 +0000 UTC m=+2.040475908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8gw7t" (UniqueName: "kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t") pod "network-check-target-5lbpx" (UID: "edffdada-2745-451b-bf87-3ff5facb7b64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 15:20:58.706912 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.706893 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45g9f\" (UniqueName: \"kubernetes.io/projected/edc388f1-89b5-45bf-9672-3e1a135fed4e-kube-api-access-45g9f\") pod \"network-metrics-daemon-kg2pm\" (UID: \"edc388f1-89b5-45bf-9672-3e1a135fed4e\") " pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:20:58.707214 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.707195 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zv28\" (UniqueName: \"kubernetes.io/projected/d8339b58-8a56-4e52-8722-52249ad123d3-kube-api-access-8zv28\") pod \"multus-84dvv\" (UID: \"d8339b58-8a56-4e52-8722-52249ad123d3\") " pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.707306 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.707288 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ttql\" (UniqueName: \"kubernetes.io/projected/3f3999dd-ca70-4115-8778-3c73e9f14124-kube-api-access-7ttql\") pod \"node-ca-km9wq\" (UID: \"3f3999dd-ca70-4115-8778-3c73e9f14124\") " pod="openshift-image-registry/node-ca-km9wq" Jul 14 15:20:58.707441 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.707423 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8trr\" (UniqueName: \"kubernetes.io/projected/9db566a2-6ba1-4a33-8ce8-3c704b556841-kube-api-access-d8trr\") pod \"tuned-8gcgt\" (UID: \"9db566a2-6ba1-4a33-8ce8-3c704b556841\") " pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.711923 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.711878 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal" event={"ID":"ebf13fbb58bd809e707ce060a950b3f9","Type":"ContainerStarted","Data":"4c9cbd241d0b2962853f254746fdb818239014cf499473c544154ce66109cbcb"} Jul 14 15:20:58.729037 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.729013 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 15:20:58.791795 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.791762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-run-systemd\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.791795 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.791793 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-log-socket\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.792059 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.791812 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5062eb6e-d839-4164-aad5-057e03ade017-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.792059 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.791846 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdtzf\" (UniqueName: \"kubernetes.io/projected/261d33cd-4df1-4f4a-92b0-fcdeece8956b-kube-api-access-zdtzf\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.792059 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.791872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae601732-7bc1-4c7b-8901-7ff3feebb722-iptables-alerter-script\") pod \"iptables-alerter-prbkb\" (UID: \"ae601732-7bc1-4c7b-8901-7ff3feebb722\") " pod="openshift-network-operator/iptables-alerter-prbkb" Jul 14 15:20:58.792059 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.791890 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-log-socket\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.792059 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.791871 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-run-systemd\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.792059 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.791894 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-systemd-units\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.792059 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.791946 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbkrl\" (UniqueName: \"kubernetes.io/projected/db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb-kube-api-access-lbkrl\") pod \"node-resolver-jj7cz\" (UID: \"db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb\") " pod="openshift-dns/node-resolver-jj7cz" Jul 14 15:20:58.792059 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.791969 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-etc-openvswitch\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.792059 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792010 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-etc-openvswitch\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.792059 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792022 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-systemd-units\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.792540 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret\") pod \"global-pull-secret-syncer-785zz\" (UID: \"f42d7bd8-f742-4b0a-83d9-2e0f121bda84\") " pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:20:58.792540 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792101 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5062eb6e-d839-4164-aad5-057e03ade017-system-cni-dir\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.792540 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792130 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5062eb6e-d839-4164-aad5-057e03ade017-cnibin\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.792540 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792151 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5062eb6e-d839-4164-aad5-057e03ade017-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.792540 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-kubelet\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.792540 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:58.792172 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 15:20:58.792540 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxdtp\" (UniqueName: \"kubernetes.io/projected/ae601732-7bc1-4c7b-8901-7ff3feebb722-kube-api-access-sxdtp\") pod \"iptables-alerter-prbkb\" (UID: \"ae601732-7bc1-4c7b-8901-7ff3feebb722\") " pod="openshift-network-operator/iptables-alerter-prbkb" Jul 14 15:20:58.792540 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792201 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5062eb6e-d839-4164-aad5-057e03ade017-system-cni-dir\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.792540 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.792540 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792219 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5062eb6e-d839-4164-aad5-057e03ade017-cnibin\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.792540 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:20:58.792251 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret podName:f42d7bd8-f742-4b0a-83d9-2e0f121bda84 nodeName:}" failed. No retries permitted until 2026-07-14 15:20:59.292228504 +0000 UTC m=+2.126755631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret") pod "global-pull-secret-syncer-785zz" (UID: "f42d7bd8-f742-4b0a-83d9-2e0f121bda84") : object "kube-system"/"original-pull-secret" not registered Jul 14 15:20:58.792540 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792248 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-kubelet\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.792540 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792274 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c16af064-1d6b-4819-b9dd-92087eab85c0-sys-fs\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.792540 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48mqx\" (UniqueName: \"kubernetes.io/projected/c16af064-1d6b-4819-b9dd-92087eab85c0-kube-api-access-48mqx\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.792540 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792311 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.792540 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-dbus\") pod \"global-pull-secret-syncer-785zz\" (UID: \"f42d7bd8-f742-4b0a-83d9-2e0f121bda84\") " pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:20:58.793278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792343 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5062eb6e-d839-4164-aad5-057e03ade017-os-release\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.793278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c16af064-1d6b-4819-b9dd-92087eab85c0-registration-dir\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.793278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792366 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c16af064-1d6b-4819-b9dd-92087eab85c0-sys-fs\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.793278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae601732-7bc1-4c7b-8901-7ff3feebb722-host-slash\") pod \"iptables-alerter-prbkb\" (UID: \"ae601732-7bc1-4c7b-8901-7ff3feebb722\") " pod="openshift-network-operator/iptables-alerter-prbkb" Jul 14 15:20:58.793278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c16af064-1d6b-4819-b9dd-92087eab85c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.793278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792417 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c16af064-1d6b-4819-b9dd-92087eab85c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.793278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.793278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792444 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5062eb6e-d839-4164-aad5-057e03ade017-os-release\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.793278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-cni-netd\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.793278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792467 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-dbus\") pod \"global-pull-secret-syncer-785zz\" (UID: \"f42d7bd8-f742-4b0a-83d9-2e0f121bda84\") " pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:20:58.793278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792473 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae601732-7bc1-4c7b-8901-7ff3feebb722-host-slash\") pod \"iptables-alerter-prbkb\" (UID: \"ae601732-7bc1-4c7b-8901-7ff3feebb722\") " pod="openshift-network-operator/iptables-alerter-prbkb" Jul 14 15:20:58.793278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792494 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-cni-netd\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.793278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792526 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.793278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792527 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c16af064-1d6b-4819-b9dd-92087eab85c0-registration-dir\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.793278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792523 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae601732-7bc1-4c7b-8901-7ff3feebb722-iptables-alerter-script\") pod \"iptables-alerter-prbkb\" (UID: \"ae601732-7bc1-4c7b-8901-7ff3feebb722\") " pod="openshift-network-operator/iptables-alerter-prbkb" Jul 14 15:20:58.793278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-kubelet-config\") pod \"global-pull-secret-syncer-785zz\" (UID: \"f42d7bd8-f742-4b0a-83d9-2e0f121bda84\") " pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:20:58.793278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792549 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c16af064-1d6b-4819-b9dd-92087eab85c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.793866 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792583 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5062eb6e-d839-4164-aad5-057e03ade017-cni-binary-copy\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.793866 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792589 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-kubelet-config\") pod \"global-pull-secret-syncer-785zz\" (UID: \"f42d7bd8-f742-4b0a-83d9-2e0f121bda84\") " pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:20:58.793866 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792574 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c16af064-1d6b-4819-b9dd-92087eab85c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.793866 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c16af064-1d6b-4819-b9dd-92087eab85c0-device-dir\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.793866 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-run-netns\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.793866 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-node-log\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.793866 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792690 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c16af064-1d6b-4819-b9dd-92087eab85c0-device-dir\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.793866 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792709 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-cni-bin\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.793866 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb-hosts-file\") pod \"node-resolver-jj7cz\" (UID: \"db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb\") " pod="openshift-dns/node-resolver-jj7cz" Jul 14 15:20:58.793866 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792755 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-node-log\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.793866 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792770 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-cni-bin\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.793866 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792773 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-run-ovn\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.793866 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792805 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-run-ovn\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.793866 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792805 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb-hosts-file\") pod \"node-resolver-jj7cz\" (UID: \"db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb\") " pod="openshift-dns/node-resolver-jj7cz" Jul 14 15:20:58.793866 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792745 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-run-netns\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.793866 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792805 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/261d33cd-4df1-4f4a-92b0-fcdeece8956b-ovnkube-config\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.793866 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792844 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5062eb6e-d839-4164-aad5-057e03ade017-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.794357 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/261d33cd-4df1-4f4a-92b0-fcdeece8956b-env-overrides\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.794357 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792915 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5062eb6e-d839-4164-aad5-057e03ade017-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.794357 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792917 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5062eb6e-d839-4164-aad5-057e03ade017-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.794357 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792960 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb-tmp-dir\") pod \"node-resolver-jj7cz\" (UID: \"db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb\") " pod="openshift-dns/node-resolver-jj7cz" Jul 14 15:20:58.794357 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.792988 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfntj\" (UniqueName: \"kubernetes.io/projected/5062eb6e-d839-4164-aad5-057e03ade017-kube-api-access-zfntj\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.794357 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.793014 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-slash\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.794357 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.793065 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5062eb6e-d839-4164-aad5-057e03ade017-cni-binary-copy\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.794357 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.793092 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-var-lib-openvswitch\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.794357 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.793121 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-host-slash\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.794357 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.793127 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/261d33cd-4df1-4f4a-92b0-fcdeece8956b-ovn-node-metrics-cert\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.794357 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.793157 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c16af064-1d6b-4819-b9dd-92087eab85c0-socket-dir\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.794357 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.793199 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-run-openvswitch\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.794357 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.793227 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/261d33cd-4df1-4f4a-92b0-fcdeece8956b-ovnkube-script-lib\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.794357 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.793259 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c16af064-1d6b-4819-b9dd-92087eab85c0-socket-dir\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.794357 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.793159 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-var-lib-openvswitch\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.794357 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.793290 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb-tmp-dir\") pod \"node-resolver-jj7cz\" (UID: \"db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb\") " pod="openshift-dns/node-resolver-jj7cz" Jul 14 15:20:58.794357 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.793303 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/261d33cd-4df1-4f4a-92b0-fcdeece8956b-run-openvswitch\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.794849 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.793443 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/261d33cd-4df1-4f4a-92b0-fcdeece8956b-env-overrides\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.794849 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.793457 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/261d33cd-4df1-4f4a-92b0-fcdeece8956b-ovnkube-config\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.794849 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.793633 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/261d33cd-4df1-4f4a-92b0-fcdeece8956b-ovnkube-script-lib\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.794849 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.793739 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5062eb6e-d839-4164-aad5-057e03ade017-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.795631 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.795611 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/261d33cd-4df1-4f4a-92b0-fcdeece8956b-ovn-node-metrics-cert\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.807774 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.807750 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48mqx\" (UniqueName: \"kubernetes.io/projected/c16af064-1d6b-4819-b9dd-92087eab85c0-kube-api-access-48mqx\") pod \"aws-ebs-csi-driver-node-qkv8s\" (UID: \"c16af064-1d6b-4819-b9dd-92087eab85c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:58.807888 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.807825 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxdtp\" (UniqueName: \"kubernetes.io/projected/ae601732-7bc1-4c7b-8901-7ff3feebb722-kube-api-access-sxdtp\") pod \"iptables-alerter-prbkb\" (UID: \"ae601732-7bc1-4c7b-8901-7ff3feebb722\") " pod="openshift-network-operator/iptables-alerter-prbkb" Jul 14 15:20:58.807939 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.807906 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbkrl\" (UniqueName: \"kubernetes.io/projected/db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb-kube-api-access-lbkrl\") pod \"node-resolver-jj7cz\" (UID: \"db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb\") " pod="openshift-dns/node-resolver-jj7cz" Jul 14 15:20:58.807976 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.807955 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdtzf\" (UniqueName: \"kubernetes.io/projected/261d33cd-4df1-4f4a-92b0-fcdeece8956b-kube-api-access-zdtzf\") pod \"ovnkube-node-g9j4v\" (UID: \"261d33cd-4df1-4f4a-92b0-fcdeece8956b\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:58.808440 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.808424 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfntj\" (UniqueName: \"kubernetes.io/projected/5062eb6e-d839-4164-aad5-057e03ade017-kube-api-access-zfntj\") pod \"multus-additional-cni-plugins-kvkjc\" (UID: \"5062eb6e-d839-4164-aad5-057e03ade017\") " pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.914399 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.914314 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-km9wq" Jul 14 15:20:58.922779 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:58.922759 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3999dd_ca70_4115_8778_3c73e9f14124.slice/crio-804fc9724911be6ce0b2da2fe6f48217f843009bf244db98ec14866dfb1f644f WatchSource:0}: Error finding container 804fc9724911be6ce0b2da2fe6f48217f843009bf244db98ec14866dfb1f644f: Status 404 returned error can't find the container with id 804fc9724911be6ce0b2da2fe6f48217f843009bf244db98ec14866dfb1f644f Jul 14 15:20:58.929278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.929262 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-89g6j" Jul 14 15:20:58.936230 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:58.936208 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b98868_c196_465a_8561_d10895ae0f3c.slice/crio-5fc1c4c901b4692047469bef9a8127c36f47ed0adf21c8c0046c8be2ec900a8c WatchSource:0}: Error finding container 5fc1c4c901b4692047469bef9a8127c36f47ed0adf21c8c0046c8be2ec900a8c: Status 404 returned error can't find the container with id 5fc1c4c901b4692047469bef9a8127c36f47ed0adf21c8c0046c8be2ec900a8c Jul 14 15:20:58.938566 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.938516 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" Jul 14 15:20:58.945396 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:58.945373 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db566a2_6ba1_4a33_8ce8_3c704b556841.slice/crio-8517c8100263bc7e6189114f0d59e92550c325106e2a9315afc45035ee72914c WatchSource:0}: Error finding container 8517c8100263bc7e6189114f0d59e92550c325106e2a9315afc45035ee72914c: Status 404 returned error can't find the container with id 8517c8100263bc7e6189114f0d59e92550c325106e2a9315afc45035ee72914c Jul 14 15:20:58.954086 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.954017 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-84dvv" Jul 14 15:20:58.961094 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:58.961077 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8339b58_8a56_4e52_8722_52249ad123d3.slice/crio-17b336a1c30fae2bfcb389779016dfe9b719315b4e18b0b34315e86bee7f33a9 WatchSource:0}: Error finding container 17b336a1c30fae2bfcb389779016dfe9b719315b4e18b0b34315e86bee7f33a9: Status 404 returned error can't find the container with id 17b336a1c30fae2bfcb389779016dfe9b719315b4e18b0b34315e86bee7f33a9 Jul 14 15:20:58.970277 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.970259 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kvkjc" Jul 14 15:20:58.977388 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:58.977367 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5062eb6e_d839_4164_aad5_057e03ade017.slice/crio-e204a0106b0a0a5bed899ef5ff7bb573adfb7939177c386d620a2c16db44a870 WatchSource:0}: Error finding container e204a0106b0a0a5bed899ef5ff7bb573adfb7939177c386d620a2c16db44a870: Status 404 returned error can't find the container with id e204a0106b0a0a5bed899ef5ff7bb573adfb7939177c386d620a2c16db44a870 Jul 14 15:20:58.987461 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.987442 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-prbkb" Jul 14 15:20:58.993950 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:58.993929 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae601732_7bc1_4c7b_8901_7ff3feebb722.slice/crio-3764c307243bb0f6be75ddd5f8334bb77d0fb2d7be65cad2e0cfd33a08202624 WatchSource:0}: Error finding container 3764c307243bb0f6be75ddd5f8334bb77d0fb2d7be65cad2e0cfd33a08202624: Status 404 returned error can't find the container with id 3764c307243bb0f6be75ddd5f8334bb77d0fb2d7be65cad2e0cfd33a08202624 Jul 14 15:20:58.998801 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:58.998786 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:20:59.004559 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:59.004529 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" Jul 14 15:20:59.005648 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:59.005620 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261d33cd_4df1_4f4a_92b0_fcdeece8956b.slice/crio-8dc501dbcf5ed52155fa0e5602635d27fd45a59089724e62ef791da46f670feb WatchSource:0}: Error finding container 8dc501dbcf5ed52155fa0e5602635d27fd45a59089724e62ef791da46f670feb: Status 404 returned error can't find the container with id 8dc501dbcf5ed52155fa0e5602635d27fd45a59089724e62ef791da46f670feb Jul 14 15:20:59.012869 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:59.012849 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16af064_1d6b_4819_b9dd_92087eab85c0.slice/crio-9f28531208ec0d94607a702a0abcf5431578a6e3acece4e7e968ccfc1290db98 WatchSource:0}: Error finding container 9f28531208ec0d94607a702a0abcf5431578a6e3acece4e7e968ccfc1290db98: Status 404 returned error can't find the container with id 9f28531208ec0d94607a702a0abcf5431578a6e3acece4e7e968ccfc1290db98 Jul 14 15:20:59.015253 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:20:59.015232 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jj7cz" Jul 14 15:20:59.021823 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:20:59.021805 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4c8e2e_aa46_4542_b5d8_8401fa2dfbeb.slice/crio-0c75ecd636826960833a0ea43d57e04676c4ee45a59c68d747badde69ec19a2a WatchSource:0}: Error finding container 0c75ecd636826960833a0ea43d57e04676c4ee45a59c68d747badde69ec19a2a: Status 404 returned error can't find the container with id 0c75ecd636826960833a0ea43d57e04676c4ee45a59c68d747badde69ec19a2a Jul 14 15:21:01.178375 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:01.178289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs\") pod \"network-metrics-daemon-kg2pm\" (UID: \"edc388f1-89b5-45bf-9672-3e1a135fed4e\") " pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:01.178511 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:01.178444 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 15:21:01.178511 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:01.178500 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs podName:edc388f1-89b5-45bf-9672-3e1a135fed4e nodeName:}" failed. No retries permitted until 2026-07-14 15:21:02.178486737 +0000 UTC m=+3.030869880 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs") pod "network-metrics-daemon-kg2pm" (UID: "edc388f1-89b5-45bf-9672-3e1a135fed4e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 15:21:01.279549 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:01.279517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gw7t\" (UniqueName: \"kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t\") pod \"network-check-target-5lbpx\" (UID: \"edffdada-2745-451b-bf87-3ff5facb7b64\") " pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:01.279675 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:01.279562 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret\") pod \"global-pull-secret-syncer-785zz\" (UID: \"f42d7bd8-f742-4b0a-83d9-2e0f121bda84\") " pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:01.279675 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:01.279672 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 15:21:01.279759 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:01.279674 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 15:21:01.279759 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:01.279694 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 15:21:01.279759 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:01.279703 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8gw7t for pod openshift-network-diagnostics/network-check-target-5lbpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 15:21:01.279759 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:01.279727 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret podName:f42d7bd8-f742-4b0a-83d9-2e0f121bda84 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:02.279706423 +0000 UTC m=+3.132089566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret") pod "global-pull-secret-syncer-785zz" (UID: "f42d7bd8-f742-4b0a-83d9-2e0f121bda84") : object "kube-system"/"original-pull-secret" not registered Jul 14 15:21:01.279759 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:01.279755 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t podName:edffdada-2745-451b-bf87-3ff5facb7b64 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:02.279743949 +0000 UTC m=+3.132127091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8gw7t" (UniqueName: "kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t") pod "network-check-target-5lbpx" (UID: "edffdada-2745-451b-bf87-3ff5facb7b64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 15:21:01.559689 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:01.559402 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 15:21:01.613652 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:01.613570 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 15:16:00 +0000 UTC" deadline="2028-02-12 02:38:18.601626912 +0000 UTC" Jul 14 15:21:01.613652 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:01.613625 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13859h17m16.988006406s" Jul 14 15:21:01.693761 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:01.693730 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:01.693908 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:01.693874 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785zz" podUID="f42d7bd8-f742-4b0a-83d9-2e0f121bda84" Jul 14 15:21:01.694036 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:01.693976 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:01.694092 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:01.694059 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5lbpx" podUID="edffdada-2745-451b-bf87-3ff5facb7b64" Jul 14 15:21:01.699742 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:01.699717 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jj7cz" event={"ID":"db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb","Type":"ContainerStarted","Data":"0c75ecd636826960833a0ea43d57e04676c4ee45a59c68d747badde69ec19a2a"} Jul 14 15:21:01.704260 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:01.704231 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" event={"ID":"c16af064-1d6b-4819-b9dd-92087eab85c0","Type":"ContainerStarted","Data":"9f28531208ec0d94607a702a0abcf5431578a6e3acece4e7e968ccfc1290db98"} Jul 14 15:21:01.707577 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:01.707552 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" event={"ID":"261d33cd-4df1-4f4a-92b0-fcdeece8956b","Type":"ContainerStarted","Data":"8dc501dbcf5ed52155fa0e5602635d27fd45a59089724e62ef791da46f670feb"} Jul 14 15:21:01.710354 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:01.710328 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-84dvv" event={"ID":"d8339b58-8a56-4e52-8722-52249ad123d3","Type":"ContainerStarted","Data":"17b336a1c30fae2bfcb389779016dfe9b719315b4e18b0b34315e86bee7f33a9"} Jul 14 15:21:01.712871 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:01.712843 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" event={"ID":"9db566a2-6ba1-4a33-8ce8-3c704b556841","Type":"ContainerStarted","Data":"8517c8100263bc7e6189114f0d59e92550c325106e2a9315afc45035ee72914c"} Jul 14 15:21:01.725853 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:01.725818 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-89g6j" event={"ID":"36b98868-c196-465a-8561-d10895ae0f3c","Type":"ContainerStarted","Data":"5fc1c4c901b4692047469bef9a8127c36f47ed0adf21c8c0046c8be2ec900a8c"} Jul 14 15:21:01.729544 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:01.729515 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-km9wq" event={"ID":"3f3999dd-ca70-4115-8778-3c73e9f14124","Type":"ContainerStarted","Data":"804fc9724911be6ce0b2da2fe6f48217f843009bf244db98ec14866dfb1f644f"} Jul 14 15:21:01.735278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:01.735237 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-prbkb" event={"ID":"ae601732-7bc1-4c7b-8901-7ff3feebb722","Type":"ContainerStarted","Data":"3764c307243bb0f6be75ddd5f8334bb77d0fb2d7be65cad2e0cfd33a08202624"} Jul 14 15:21:01.747676 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:01.747638 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvkjc" event={"ID":"5062eb6e-d839-4164-aad5-057e03ade017","Type":"ContainerStarted","Data":"e204a0106b0a0a5bed899ef5ff7bb573adfb7939177c386d620a2c16db44a870"} Jul 14 15:21:02.186291 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:02.186249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs\") pod \"network-metrics-daemon-kg2pm\" (UID: \"edc388f1-89b5-45bf-9672-3e1a135fed4e\") " pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:02.186481 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:02.186427 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 15:21:02.186557 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:02.186509 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs podName:edc388f1-89b5-45bf-9672-3e1a135fed4e nodeName:}" failed. No retries permitted until 2026-07-14 15:21:04.186487636 +0000 UTC m=+5.038870803 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs") pod "network-metrics-daemon-kg2pm" (UID: "edc388f1-89b5-45bf-9672-3e1a135fed4e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 15:21:02.289255 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:02.288939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gw7t\" (UniqueName: \"kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t\") pod \"network-check-target-5lbpx\" (UID: \"edffdada-2745-451b-bf87-3ff5facb7b64\") " pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:02.289255 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:02.289017 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret\") pod \"global-pull-secret-syncer-785zz\" (UID: \"f42d7bd8-f742-4b0a-83d9-2e0f121bda84\") " pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:02.289255 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:02.289110 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 15:21:02.289255 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:02.289143 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 15:21:02.289255 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:02.289158 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8gw7t for pod openshift-network-diagnostics/network-check-target-5lbpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 15:21:02.289255 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:02.289186 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 15:21:02.289255 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:02.289243 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t podName:edffdada-2745-451b-bf87-3ff5facb7b64 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:04.289224493 +0000 UTC m=+5.141607636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8gw7t" (UniqueName: "kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t") pod "network-check-target-5lbpx" (UID: "edffdada-2745-451b-bf87-3ff5facb7b64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 15:21:02.289255 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:02.289266 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret podName:f42d7bd8-f742-4b0a-83d9-2e0f121bda84 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:04.289258101 +0000 UTC m=+5.141641244 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret") pod "global-pull-secret-syncer-785zz" (UID: "f42d7bd8-f742-4b0a-83d9-2e0f121bda84") : object "kube-system"/"original-pull-secret" not registered Jul 14 15:21:02.614248 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:02.614118 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 15:16:00 +0000 UTC" deadline="2028-04-07 06:17:02.597259702 +0000 UTC" Jul 14 15:21:02.614248 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:02.614197 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15182h55m59.983067743s" Jul 14 15:21:02.691072 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:02.691011 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:02.691263 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:02.691157 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kg2pm" podUID="edc388f1-89b5-45bf-9672-3e1a135fed4e" Jul 14 15:21:03.692097 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:03.691743 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:03.692097 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:03.691881 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785zz" podUID="f42d7bd8-f742-4b0a-83d9-2e0f121bda84" Jul 14 15:21:03.692598 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:03.692291 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:03.692598 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:03.692379 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5lbpx" podUID="edffdada-2745-451b-bf87-3ff5facb7b64" Jul 14 15:21:04.206440 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:04.206312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs\") pod \"network-metrics-daemon-kg2pm\" (UID: \"edc388f1-89b5-45bf-9672-3e1a135fed4e\") " pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:04.206606 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:04.206465 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 15:21:04.206606 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:04.206535 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs podName:edc388f1-89b5-45bf-9672-3e1a135fed4e nodeName:}" failed. No retries permitted until 2026-07-14 15:21:08.206515681 +0000 UTC m=+9.058898830 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs") pod "network-metrics-daemon-kg2pm" (UID: "edc388f1-89b5-45bf-9672-3e1a135fed4e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 15:21:04.307678 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:04.307642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gw7t\" (UniqueName: \"kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t\") pod \"network-check-target-5lbpx\" (UID: \"edffdada-2745-451b-bf87-3ff5facb7b64\") " pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:04.307862 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:04.307703 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret\") pod \"global-pull-secret-syncer-785zz\" (UID: \"f42d7bd8-f742-4b0a-83d9-2e0f121bda84\") " pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:04.307862 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:04.307830 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 15:21:04.307971 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:04.307896 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret podName:f42d7bd8-f742-4b0a-83d9-2e0f121bda84 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:08.307877659 +0000 UTC m=+9.160260808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret") pod "global-pull-secret-syncer-785zz" (UID: "f42d7bd8-f742-4b0a-83d9-2e0f121bda84") : object "kube-system"/"original-pull-secret" not registered Jul 14 15:21:04.308036 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:04.307993 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 15:21:04.308036 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:04.308011 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 15:21:04.308036 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:04.308028 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8gw7t for pod openshift-network-diagnostics/network-check-target-5lbpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 15:21:04.308197 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:04.308069 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t podName:edffdada-2745-451b-bf87-3ff5facb7b64 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:08.3080561 +0000 UTC m=+9.160439246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8gw7t" (UniqueName: "kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t") pod "network-check-target-5lbpx" (UID: "edffdada-2745-451b-bf87-3ff5facb7b64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 15:21:04.692255 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:04.690856 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:04.692255 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:04.690997 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kg2pm" podUID="edc388f1-89b5-45bf-9672-3e1a135fed4e" Jul 14 15:21:05.692802 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:05.692280 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:05.692802 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:05.692418 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5lbpx" podUID="edffdada-2745-451b-bf87-3ff5facb7b64" Jul 14 15:21:05.692802 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:05.692529 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:05.692802 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:05.692648 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785zz" podUID="f42d7bd8-f742-4b0a-83d9-2e0f121bda84" Jul 14 15:21:06.691117 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:06.691082 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:06.691328 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:06.691261 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kg2pm" podUID="edc388f1-89b5-45bf-9672-3e1a135fed4e" Jul 14 15:21:07.693381 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:07.693344 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:07.693819 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:07.693546 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5lbpx" podUID="edffdada-2745-451b-bf87-3ff5facb7b64" Jul 14 15:21:07.694089 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:07.694068 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:07.694281 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:07.694236 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785zz" podUID="f42d7bd8-f742-4b0a-83d9-2e0f121bda84" Jul 14 15:21:08.242457 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:08.242417 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs\") pod \"network-metrics-daemon-kg2pm\" (UID: \"edc388f1-89b5-45bf-9672-3e1a135fed4e\") " pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:08.242637 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:08.242584 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 15:21:08.242705 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:08.242692 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs podName:edc388f1-89b5-45bf-9672-3e1a135fed4e nodeName:}" failed. No retries permitted until 2026-07-14 15:21:16.242674557 +0000 UTC m=+17.095057699 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs") pod "network-metrics-daemon-kg2pm" (UID: "edc388f1-89b5-45bf-9672-3e1a135fed4e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 15:21:08.342835 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:08.342796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gw7t\" (UniqueName: \"kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t\") pod \"network-check-target-5lbpx\" (UID: \"edffdada-2745-451b-bf87-3ff5facb7b64\") " pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:08.342835 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:08.342849 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret\") pod \"global-pull-secret-syncer-785zz\" (UID: \"f42d7bd8-f742-4b0a-83d9-2e0f121bda84\") " pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:08.343080 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:08.342948 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 15:21:08.343080 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:08.342985 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 15:21:08.343080 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:08.343012 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 15:21:08.343080 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:08.343025 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8gw7t for pod openshift-network-diagnostics/network-check-target-5lbpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 15:21:08.343080 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:08.342996 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret podName:f42d7bd8-f742-4b0a-83d9-2e0f121bda84 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:16.342982824 +0000 UTC m=+17.195365970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret") pod "global-pull-secret-syncer-785zz" (UID: "f42d7bd8-f742-4b0a-83d9-2e0f121bda84") : object "kube-system"/"original-pull-secret" not registered Jul 14 15:21:08.343345 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:08.343092 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t podName:edffdada-2745-451b-bf87-3ff5facb7b64 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:16.343076028 +0000 UTC m=+17.195459170 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8gw7t" (UniqueName: "kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t") pod "network-check-target-5lbpx" (UID: "edffdada-2745-451b-bf87-3ff5facb7b64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 15:21:08.691261 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:08.691185 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:08.691425 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:08.691324 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kg2pm" podUID="edc388f1-89b5-45bf-9672-3e1a135fed4e" Jul 14 15:21:09.691508 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:09.691473 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:09.691978 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:09.691571 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785zz" podUID="f42d7bd8-f742-4b0a-83d9-2e0f121bda84" Jul 14 15:21:09.691978 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:09.691921 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:09.692103 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:09.692020 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5lbpx" podUID="edffdada-2745-451b-bf87-3ff5facb7b64" Jul 14 15:21:10.690705 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:10.690663 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:10.690896 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:10.690798 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kg2pm" podUID="edc388f1-89b5-45bf-9672-3e1a135fed4e" Jul 14 15:21:11.691530 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:11.691499 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:11.691981 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:11.691498 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:11.691981 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:11.691644 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785zz" podUID="f42d7bd8-f742-4b0a-83d9-2e0f121bda84" Jul 14 15:21:11.691981 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:11.691699 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5lbpx" podUID="edffdada-2745-451b-bf87-3ff5facb7b64" Jul 14 15:21:12.691456 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:12.691227 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:12.691634 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:12.691576 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kg2pm" podUID="edc388f1-89b5-45bf-9672-3e1a135fed4e" Jul 14 15:21:13.690926 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:13.690877 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:13.691102 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:13.691012 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785zz" podUID="f42d7bd8-f742-4b0a-83d9-2e0f121bda84" Jul 14 15:21:13.691102 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:13.691069 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:13.691301 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:13.691201 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5lbpx" podUID="edffdada-2745-451b-bf87-3ff5facb7b64" Jul 14 15:21:14.690825 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:14.690789 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:14.691251 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:14.690911 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kg2pm" podUID="edc388f1-89b5-45bf-9672-3e1a135fed4e" Jul 14 15:21:15.691115 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:15.691080 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:15.691582 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:15.691237 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785zz" podUID="f42d7bd8-f742-4b0a-83d9-2e0f121bda84" Jul 14 15:21:15.691582 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:15.691280 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:15.691582 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:15.691391 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5lbpx" podUID="edffdada-2745-451b-bf87-3ff5facb7b64" Jul 14 15:21:16.301682 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:16.301650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs\") pod \"network-metrics-daemon-kg2pm\" (UID: \"edc388f1-89b5-45bf-9672-3e1a135fed4e\") " pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:16.301914 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:16.301764 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 15:21:16.301914 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:16.301827 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs podName:edc388f1-89b5-45bf-9672-3e1a135fed4e nodeName:}" failed. No retries permitted until 2026-07-14 15:21:32.301809264 +0000 UTC m=+33.154192409 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs") pod "network-metrics-daemon-kg2pm" (UID: "edc388f1-89b5-45bf-9672-3e1a135fed4e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 15:21:16.402016 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:16.401982 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gw7t\" (UniqueName: \"kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t\") pod \"network-check-target-5lbpx\" (UID: \"edffdada-2745-451b-bf87-3ff5facb7b64\") " pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:16.402187 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:16.402039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret\") pod \"global-pull-secret-syncer-785zz\" (UID: \"f42d7bd8-f742-4b0a-83d9-2e0f121bda84\") " pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:16.402187 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:16.402164 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 15:21:16.402258 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:16.402200 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 15:21:16.402258 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:16.402210 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8gw7t for pod openshift-network-diagnostics/network-check-target-5lbpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 15:21:16.402258 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:16.402221 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 15:21:16.402349 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:16.402261 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t podName:edffdada-2745-451b-bf87-3ff5facb7b64 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:32.402247699 +0000 UTC m=+33.254630843 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8gw7t" (UniqueName: "kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t") pod "network-check-target-5lbpx" (UID: "edffdada-2745-451b-bf87-3ff5facb7b64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 15:21:16.402349 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:16.402280 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret podName:f42d7bd8-f742-4b0a-83d9-2e0f121bda84 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:32.402269713 +0000 UTC m=+33.254652857 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret") pod "global-pull-secret-syncer-785zz" (UID: "f42d7bd8-f742-4b0a-83d9-2e0f121bda84") : object "kube-system"/"original-pull-secret" not registered Jul 14 15:21:16.691163 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:16.691084 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:16.691565 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:16.691248 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kg2pm" podUID="edc388f1-89b5-45bf-9672-3e1a135fed4e" Jul 14 15:21:17.691628 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:17.691590 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:17.692113 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:17.691639 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:17.692113 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:17.691731 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785zz" podUID="f42d7bd8-f742-4b0a-83d9-2e0f121bda84" Jul 14 15:21:17.692113 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:17.691821 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5lbpx" podUID="edffdada-2745-451b-bf87-3ff5facb7b64" Jul 14 15:21:18.691336 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:18.691296 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:18.691492 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:18.691423 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kg2pm" podUID="edc388f1-89b5-45bf-9672-3e1a135fed4e" Jul 14 15:21:19.691517 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:19.691470 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:19.691953 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:19.691591 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785zz" podUID="f42d7bd8-f742-4b0a-83d9-2e0f121bda84" Jul 14 15:21:19.691953 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:19.691672 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:19.691953 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:19.691773 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5lbpx" podUID="edffdada-2745-451b-bf87-3ff5facb7b64" Jul 14 15:21:20.690874 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:20.690849 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:20.690978 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:20.690960 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kg2pm" podUID="edc388f1-89b5-45bf-9672-3e1a135fed4e" Jul 14 15:21:20.801022 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:20.800938 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" event={"ID":"83a908925c0c8a87df4dd7fc6977e74c","Type":"ContainerStarted","Data":"ed174902aec495962e4403be547d4baba5ab03c2a3965a407ec538a71a404e59"} Jul 14 15:21:20.809021 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:20.808965 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal" event={"ID":"ebf13fbb58bd809e707ce060a950b3f9","Type":"ContainerStarted","Data":"b1b6408ee3944d297dfaddfb0e1a684dbe8f79b780fc0fdc47668e8f755e0046"} Jul 14 15:21:20.828678 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:20.828473 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal" podStartSLOduration=20.828457044 podStartE2EDuration="20.828457044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:20.827911615 +0000 UTC m=+21.680294784" watchObservedRunningTime="2026-07-14 15:21:20.828457044 +0000 UTC m=+21.680840208" Jul 14 15:21:21.691444 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:21.691152 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:21.691593 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:21.691512 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785zz" podUID="f42d7bd8-f742-4b0a-83d9-2e0f121bda84" Jul 14 15:21:21.692012 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:21.691902 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:21.692207 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:21.692162 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5lbpx" podUID="edffdada-2745-451b-bf87-3ff5facb7b64" Jul 14 15:21:21.813252 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:21.813216 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" event={"ID":"c16af064-1d6b-4819-b9dd-92087eab85c0","Type":"ContainerStarted","Data":"8f3b0d50e9c8d58e881546edd246f39cb0d5b8a07aa78ad761f2ebf544795962"} Jul 14 15:21:21.815818 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:21.815790 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" event={"ID":"261d33cd-4df1-4f4a-92b0-fcdeece8956b","Type":"ContainerStarted","Data":"647b77ba3668c42758144fafaf1cebf195e12b46a8e3bc14f57bf222aa69a6ed"} Jul 14 15:21:21.815949 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:21.815872 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" event={"ID":"261d33cd-4df1-4f4a-92b0-fcdeece8956b","Type":"ContainerStarted","Data":"3c1866dcaefea3a98865890a2687e89e7f5f59739ca54a13218cdac8d4b5cfa6"} Jul 14 15:21:21.817575 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:21.817547 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-84dvv" event={"ID":"d8339b58-8a56-4e52-8722-52249ad123d3","Type":"ContainerStarted","Data":"b7d7fcf7532429db6a6d80b99adbad8a07d204235f608a1eef977add1d44de5a"} Jul 14 15:21:21.819441 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:21.819415 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" event={"ID":"9db566a2-6ba1-4a33-8ce8-3c704b556841","Type":"ContainerStarted","Data":"31a3245a4e363e488bdf635ea92ba1e26b3a08fb3ae3ad171377871300f9cd75"} Jul 14 15:21:21.828709 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:21.828682 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-89g6j" event={"ID":"36b98868-c196-465a-8561-d10895ae0f3c","Type":"ContainerStarted","Data":"a73f456aff0ed7c4229289a23dae7156203d8b7d25b1d4a68313c7c0c863ca86"} Jul 14 15:21:21.831370 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:21.831340 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-km9wq" event={"ID":"3f3999dd-ca70-4115-8778-3c73e9f14124","Type":"ContainerStarted","Data":"e0e40d2f1ec5f4eed6042fe45aff440a38eb5295d5cf4f132735244f77d9bad3"} Jul 14 15:21:21.833073 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:21.833037 2579 generic.go:356] "Generic (PLEG): container finished" podID="5062eb6e-d839-4164-aad5-057e03ade017" containerID="770e479f39c6dfd47df1b9f5fc286fd1dabd256191c06241f420e38ff4a4ca92" exitCode=0 Jul 14 15:21:21.833186 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:21.833118 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvkjc" event={"ID":"5062eb6e-d839-4164-aad5-057e03ade017","Type":"ContainerDied","Data":"770e479f39c6dfd47df1b9f5fc286fd1dabd256191c06241f420e38ff4a4ca92"} Jul 14 15:21:21.835398 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:21.835286 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jj7cz" event={"ID":"db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb","Type":"ContainerStarted","Data":"2b5d6776b34b904e35cc078b681fa9727ba8232ac3eff97d46fcd991cf1fc5fa"} Jul 14 15:21:21.848363 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:21.848299 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-84dvv" podStartSLOduration=2.997606855 podStartE2EDuration="22.848280572s" totalImagesPullingTime="19.850673717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:21.847758533 +0000 UTC m=+22.700141698" watchObservedRunningTime="2026-07-14 15:21:21.848280572 +0000 UTC m=+22.700663741" Jul 14 15:21:21.904245 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:21.904201 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-km9wq" podStartSLOduration=3.269675725 podStartE2EDuration="22.904185264s" totalImagesPullingTime="19.634509539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:21.866463739 +0000 UTC m=+22.718846903" watchObservedRunningTime="2026-07-14 15:21:21.904185264 +0000 UTC m=+22.756568424" Jul 14 15:21:21.904492 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:21.904467 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8gcgt" podStartSLOduration=3.258633391 podStartE2EDuration="22.904458435s" totalImagesPullingTime="19.645825044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:21.90431339 +0000 UTC m=+22.756696556" watchObservedRunningTime="2026-07-14 15:21:21.904458435 +0000 UTC m=+22.756841605" Jul 14 15:21:21.929058 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:21.929021 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jj7cz" podStartSLOduration=2.393067636 podStartE2EDuration="21.929010508s" totalImagesPullingTime="19.535942872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:21:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:21.928890065 +0000 UTC m=+22.781273229" watchObservedRunningTime="2026-07-14 15:21:21.929010508 +0000 UTC m=+22.781393673" Jul 14 15:21:21.981369 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:21.981325 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-89g6j" podStartSLOduration=3.360366694 podStartE2EDuration="22.981308987s" totalImagesPullingTime="19.620942293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:21.956746571 +0000 UTC m=+22.809129737" watchObservedRunningTime="2026-07-14 15:21:21.981308987 +0000 UTC m=+22.833692153" Jul 14 15:21:22.691743 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:22.691208 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:22.691743 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:22.691470 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kg2pm" podUID="edc388f1-89b5-45bf-9672-3e1a135fed4e" Jul 14 15:21:22.812348 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:22.812308 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 14 15:21:22.839747 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:22.839722 2579 generic.go:356] "Generic (PLEG): container finished" podID="83a908925c0c8a87df4dd7fc6977e74c" containerID="b6523bfdf6607cc1e046cafcc3c075400aa647747edeea23bdd199c7eab19873" exitCode=0 Jul 14 15:21:22.840071 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:22.839815 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" event={"ID":"83a908925c0c8a87df4dd7fc6977e74c","Type":"ContainerDied","Data":"b6523bfdf6607cc1e046cafcc3c075400aa647747edeea23bdd199c7eab19873"} Jul 14 15:21:22.841510 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:22.841487 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-prbkb" event={"ID":"ae601732-7bc1-4c7b-8901-7ff3feebb722","Type":"ContainerStarted","Data":"ece7a1f43adb3ea86404d73a83540c063aeee1df8f0f1dacba1ca25b759f296b"} Jul 14 15:21:22.843528 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:22.843496 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" event={"ID":"c16af064-1d6b-4819-b9dd-92087eab85c0","Type":"ContainerStarted","Data":"165983f232d4cecf6cbed0daef536550265d55d10a5728ceab6b7662eda68838"} Jul 14 15:21:22.845631 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:22.845607 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" event={"ID":"261d33cd-4df1-4f4a-92b0-fcdeece8956b","Type":"ContainerStarted","Data":"de4ddd12b9f1ca88d999f9b9e4a011a1da7ecbdfa9122761546445e50a76a689"} Jul 14 15:21:23.295079 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:23.294996 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-89g6j" Jul 14 15:21:23.295742 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:23.295721 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-89g6j" Jul 14 15:21:23.311998 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:23.311944 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-prbkb" podStartSLOduration=3.748167612 podStartE2EDuration="23.311924151s" totalImagesPullingTime="19.563756539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:21:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:22.856984176 +0000 UTC m=+23.709367341" watchObservedRunningTime="2026-07-14 15:21:23.311924151 +0000 UTC m=+24.164307317" Jul 14 15:21:23.635826 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:23.635661 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T15:21:22.812328791Z","UUID":"a7f4931c-f6bf-4aac-a6e2-2c824cf5d942","Handler":null,"Name":"","Endpoint":""} Jul 14 15:21:23.637467 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:23.637437 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 14 15:21:23.637603 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:23.637480 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 14 15:21:23.691524 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:23.691497 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:23.691672 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:23.691608 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785zz" podUID="f42d7bd8-f742-4b0a-83d9-2e0f121bda84" Jul 14 15:21:23.691928 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:23.691912 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:23.692008 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:23.691983 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5lbpx" podUID="edffdada-2745-451b-bf87-3ff5facb7b64" Jul 14 15:21:23.852989 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:23.852129 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" event={"ID":"261d33cd-4df1-4f4a-92b0-fcdeece8956b","Type":"ContainerStarted","Data":"4cc58e8aefe059ac06e726fc464204109c6df5220b6732de02d4698497b0dd45"} Jul 14 15:21:23.852989 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:23.852237 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" event={"ID":"261d33cd-4df1-4f4a-92b0-fcdeece8956b","Type":"ContainerStarted","Data":"28d1c85db03b68ccd9b0e790b96f2b34c6d376fc4f41b0798f32a1c947356c9a"} Jul 14 15:21:23.852989 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:23.852283 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" event={"ID":"261d33cd-4df1-4f4a-92b0-fcdeece8956b","Type":"ContainerStarted","Data":"03dc17c21df83d633ac2e42e56185d80125840b08c89a802443a4a11a65d1b1d"} Jul 14 15:21:23.852989 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:23.852945 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-89g6j" Jul 14 15:21:23.854130 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:23.854111 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-89g6j" Jul 14 15:21:24.691245 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:24.691165 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:24.691400 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:24.691288 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kg2pm" podUID="edc388f1-89b5-45bf-9672-3e1a135fed4e" Jul 14 15:21:24.856687 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:24.856646 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" event={"ID":"c16af064-1d6b-4819-b9dd-92087eab85c0","Type":"ContainerStarted","Data":"fa0169fa0fcacb113a0f523b2827fc024ee40a37aa09cd47261ee6bf96a3dbd1"} Jul 14 15:21:24.858938 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:24.858908 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" event={"ID":"83a908925c0c8a87df4dd7fc6977e74c","Type":"ContainerStarted","Data":"10595d799e4c63b4a1f51666dffd810c9b9211691da916eced636618753f1b95"} Jul 14 15:21:24.895429 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:24.895380 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" podStartSLOduration=24.895362693 podStartE2EDuration="24.895362693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:24.895353552 +0000 UTC m=+25.747736716" watchObservedRunningTime="2026-07-14 15:21:24.895362693 +0000 UTC m=+25.747745859" Jul 14 15:21:24.895905 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:24.895873 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkv8s" podStartSLOduration=2.278214209 podStartE2EDuration="24.895860411s" totalImagesPullingTime="22.617646202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:21:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:24.877643073 +0000 UTC m=+25.730026287" watchObservedRunningTime="2026-07-14 15:21:24.895860411 +0000 UTC m=+25.748243576" Jul 14 15:21:25.691147 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:25.691110 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:25.691345 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:25.691155 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:25.691345 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:25.691268 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785zz" podUID="f42d7bd8-f742-4b0a-83d9-2e0f121bda84" Jul 14 15:21:25.691455 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:25.691387 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5lbpx" podUID="edffdada-2745-451b-bf87-3ff5facb7b64" Jul 14 15:21:26.691104 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:26.691025 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:26.691490 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:26.691133 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kg2pm" podUID="edc388f1-89b5-45bf-9672-3e1a135fed4e" Jul 14 15:21:26.866871 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:26.866823 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" event={"ID":"261d33cd-4df1-4f4a-92b0-fcdeece8956b","Type":"ContainerStarted","Data":"b9d491e40c29f4ea26c266d78b6c8e2c97a40ba664902fce833ed8cce7245a48"} Jul 14 15:21:26.868561 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:26.868531 2579 generic.go:356] "Generic (PLEG): container finished" podID="5062eb6e-d839-4164-aad5-057e03ade017" containerID="f1da8e6008b7afe079e9782171be1a74bc2952c57bdf90eb77ceb0e30dce6cf3" exitCode=0 Jul 14 15:21:26.868705 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:26.868578 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvkjc" event={"ID":"5062eb6e-d839-4164-aad5-057e03ade017","Type":"ContainerDied","Data":"f1da8e6008b7afe079e9782171be1a74bc2952c57bdf90eb77ceb0e30dce6cf3"} Jul 14 15:21:27.691212 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:27.691024 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:27.691639 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:27.691023 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:27.691639 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:27.691292 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785zz" podUID="f42d7bd8-f742-4b0a-83d9-2e0f121bda84" Jul 14 15:21:27.691639 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:27.691381 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5lbpx" podUID="edffdada-2745-451b-bf87-3ff5facb7b64" Jul 14 15:21:28.691131 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:28.691047 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:28.691278 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:28.691208 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kg2pm" podUID="edc388f1-89b5-45bf-9672-3e1a135fed4e" Jul 14 15:21:28.876513 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:28.876474 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" event={"ID":"261d33cd-4df1-4f4a-92b0-fcdeece8956b","Type":"ContainerStarted","Data":"5e78f16f4dad3c662ec94cd58c931b6e362fddc336f002bfa9ae04348fe2a6a7"} Jul 14 15:21:28.876796 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:28.876748 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:21:28.876906 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:28.876827 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:21:28.878975 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:28.878952 2579 generic.go:356] "Generic (PLEG): container finished" podID="5062eb6e-d839-4164-aad5-057e03ade017" containerID="5a3a31104f9af2610a025792d9c6dbde672accb1493447203a4b1290a4eb61ce" exitCode=0 Jul 14 15:21:28.879055 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:28.878986 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvkjc" event={"ID":"5062eb6e-d839-4164-aad5-057e03ade017","Type":"ContainerDied","Data":"5a3a31104f9af2610a025792d9c6dbde672accb1493447203a4b1290a4eb61ce"} Jul 14 15:21:28.893358 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:28.893338 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:21:28.905825 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:28.905788 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" podStartSLOduration=7.593753377 podStartE2EDuration="28.905775568s" totalImagesPullingTime="21.312022191s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:21:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:28.905634413 +0000 UTC m=+29.758017580" watchObservedRunningTime="2026-07-14 15:21:28.905775568 +0000 UTC m=+29.758158732" Jul 14 15:21:29.691758 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:29.691721 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:29.692347 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:29.691832 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785zz" podUID="f42d7bd8-f742-4b0a-83d9-2e0f121bda84" Jul 14 15:21:29.692347 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:29.691876 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:29.692347 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:29.691966 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5lbpx" podUID="edffdada-2745-451b-bf87-3ff5facb7b64" Jul 14 15:21:29.884449 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:29.884418 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:21:29.899727 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:29.899704 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:21:30.493696 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:30.493528 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-785zz"] Jul 14 15:21:30.493871 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:30.493797 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:30.493939 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:30.493903 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785zz" podUID="f42d7bd8-f742-4b0a-83d9-2e0f121bda84" Jul 14 15:21:30.495965 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:30.495942 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5lbpx"] Jul 14 15:21:30.496080 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:30.496036 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:30.496131 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:30.496114 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5lbpx" podUID="edffdada-2745-451b-bf87-3ff5facb7b64" Jul 14 15:21:30.496794 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:30.496769 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kg2pm"] Jul 14 15:21:30.496927 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:30.496881 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:30.497032 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:30.497000 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kg2pm" podUID="edc388f1-89b5-45bf-9672-3e1a135fed4e" Jul 14 15:21:31.890249 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:31.890209 2579 generic.go:356] "Generic (PLEG): container finished" podID="5062eb6e-d839-4164-aad5-057e03ade017" containerID="68b7eb07fe37eb525200ee7646ef0ca145d231763b5519efff1849bba3cebf59" exitCode=0 Jul 14 15:21:31.890626 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:31.890271 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvkjc" event={"ID":"5062eb6e-d839-4164-aad5-057e03ade017","Type":"ContainerDied","Data":"68b7eb07fe37eb525200ee7646ef0ca145d231763b5519efff1849bba3cebf59"} Jul 14 15:21:32.326182 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:32.326143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs\") pod \"network-metrics-daemon-kg2pm\" (UID: \"edc388f1-89b5-45bf-9672-3e1a135fed4e\") " pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:32.326348 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:32.326289 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 15:21:32.326392 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:32.326353 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs podName:edc388f1-89b5-45bf-9672-3e1a135fed4e nodeName:}" failed. No retries permitted until 2026-07-14 15:22:04.32633741 +0000 UTC m=+65.178720553 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs") pod "network-metrics-daemon-kg2pm" (UID: "edc388f1-89b5-45bf-9672-3e1a135fed4e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 15:21:32.426916 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:32.426887 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gw7t\" (UniqueName: \"kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t\") pod \"network-check-target-5lbpx\" (UID: \"edffdada-2745-451b-bf87-3ff5facb7b64\") " pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:32.427057 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:32.426926 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret\") pod \"global-pull-secret-syncer-785zz\" (UID: \"f42d7bd8-f742-4b0a-83d9-2e0f121bda84\") " pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:32.427057 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:32.427018 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 15:21:32.427131 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:32.427051 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 15:21:32.427131 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:32.427072 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret podName:f42d7bd8-f742-4b0a-83d9-2e0f121bda84 nodeName:}" failed. No retries permitted until 2026-07-14 15:22:04.427058066 +0000 UTC m=+65.279441209 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret") pod "global-pull-secret-syncer-785zz" (UID: "f42d7bd8-f742-4b0a-83d9-2e0f121bda84") : object "kube-system"/"original-pull-secret" not registered Jul 14 15:21:32.427131 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:32.427072 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 15:21:32.427131 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:32.427086 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8gw7t for pod openshift-network-diagnostics/network-check-target-5lbpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 15:21:32.427131 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:32.427128 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t podName:edffdada-2745-451b-bf87-3ff5facb7b64 nodeName:}" failed. No retries permitted until 2026-07-14 15:22:04.427112937 +0000 UTC m=+65.279496080 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8gw7t" (UniqueName: "kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t") pod "network-check-target-5lbpx" (UID: "edffdada-2745-451b-bf87-3ff5facb7b64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 15:21:32.690943 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:32.690864 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:32.691091 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:32.690892 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:32.691091 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:32.691054 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5lbpx" podUID="edffdada-2745-451b-bf87-3ff5facb7b64" Jul 14 15:21:32.691091 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:32.690893 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:32.691240 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:32.691108 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kg2pm" podUID="edc388f1-89b5-45bf-9672-3e1a135fed4e" Jul 14 15:21:32.691240 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:32.691219 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-785zz" podUID="f42d7bd8-f742-4b0a-83d9-2e0f121bda84" Jul 14 15:21:34.497039 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.496961 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeReady" Jul 14 15:21:34.497463 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.497108 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 14 15:21:34.545482 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.545437 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2zm28"] Jul 14 15:21:34.548989 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.548970 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l52n9"] Jul 14 15:21:34.549105 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.549089 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2zm28" Jul 14 15:21:34.552334 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.552313 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l52n9" Jul 14 15:21:34.552548 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.552510 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 14 15:21:34.552548 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.552532 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 14 15:21:34.552548 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.552542 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 14 15:21:34.552740 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.552551 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 14 15:21:34.552740 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.552560 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9krcw\"" Jul 14 15:21:34.555087 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.555064 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 14 15:21:34.555218 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.555144 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 14 15:21:34.556012 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.555993 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-npbnt\"" Jul 14 15:21:34.585155 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.584969 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-b9cq8"] Jul 14 15:21:34.589341 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.589318 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jkhlz"] Jul 14 15:21:34.594511 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.590842 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-b9cq8" Jul 14 15:21:34.595115 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.595097 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2zm28"] Jul 14 15:21:34.595235 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.595137 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l52n9"] Jul 14 15:21:34.595235 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.595163 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdm4"] Jul 14 15:21:34.595355 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.595241 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jkhlz" Jul 14 15:21:34.596531 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.595785 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-4lpkn\"" Jul 14 15:21:34.597452 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.597436 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 14 15:21:34.597652 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.597630 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 14 15:21:34.598067 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.597861 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 14 15:21:34.598067 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.597891 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 14 15:21:34.598650 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.598630 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 14 15:21:34.598763 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.598744 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 14 15:21:34.598836 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.598766 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 14 15:21:34.598836 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.598632 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nwgqd\"" Jul 14 15:21:34.598836 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.598797 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 14 15:21:34.599499 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.599480 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-4xn55"] Jul 14 15:21:34.599633 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.599614 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdm4" Jul 14 15:21:34.603367 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.603323 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 14 15:21:34.603496 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.603388 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4x79w\"" Jul 14 15:21:34.603586 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.603329 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 14 15:21:34.603642 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.603612 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 14 15:21:34.603699 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.603612 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 14 15:21:34.603777 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.603758 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6cd7778556-bmkdk"] Jul 14 15:21:34.603941 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.603926 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-4xn55" Jul 14 15:21:34.608500 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.607941 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 14 15:21:34.609360 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.609332 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-wnzh4\"" Jul 14 15:21:34.609581 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.609562 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 14 15:21:34.609861 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.609843 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bf4cb94c7-wmnzm"] Jul 14 15:21:34.610497 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.610471 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:34.612798 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.612779 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-zxnts"] Jul 14 15:21:34.613594 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.613344 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.613594 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.613486 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-n6jhf\"" Jul 14 15:21:34.613594 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.613531 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 15:21:34.615719 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.615083 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 14 15:21:34.615719 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.615307 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 14 15:21:34.615719 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.615495 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 14 15:21:34.615719 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.615508 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 15:21:34.615719 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.615548 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 14 15:21:34.616253 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.616158 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 14 15:21:34.616253 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.616196 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 14 15:21:34.616544 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.616321 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 14 15:21:34.616544 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.616365 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-twn9v\"" Jul 14 15:21:34.618607 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.618582 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh"] Jul 14 15:21:34.618934 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.618914 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" Jul 14 15:21:34.621993 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.621954 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-b9cq8"] Jul 14 15:21:34.622089 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.622005 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xrzdx"] Jul 14 15:21:34.622089 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.622082 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 14 15:21:34.622245 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.622190 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wz4th\"" Jul 14 15:21:34.622297 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.622264 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 14 15:21:34.622349 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.622337 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 14 15:21:34.622586 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.622568 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 14 15:21:34.622689 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.622600 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 14 15:21:34.622750 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.622713 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh" Jul 14 15:21:34.626384 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.626363 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-4xn55"] Jul 14 15:21:34.626527 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.626516 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-t4h7n"] Jul 14 15:21:34.626837 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.626817 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 14 15:21:34.627139 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.627121 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-48k9t\"" Jul 14 15:21:34.627367 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.627348 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 14 15:21:34.627566 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.627547 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 14 15:21:34.627715 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.627702 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 14 15:21:34.627886 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.627872 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xrzdx" Jul 14 15:21:34.630729 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.630694 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdm4"] Jul 14 15:21:34.630824 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.630762 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lznzq"] Jul 14 15:21:34.630931 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.630913 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p6wcc\"" Jul 14 15:21:34.631164 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.631145 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 14 15:21:34.631387 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.631368 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-t4h7n" Jul 14 15:21:34.631770 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.631655 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 14 15:21:34.631979 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.631960 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 14 15:21:34.635050 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.635026 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 15:21:34.635373 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.635336 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-2blb6\"" Jul 14 15:21:34.635550 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.635533 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 15:21:34.635924 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.635903 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 14 15:21:34.637849 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.637830 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6cd7778556-bmkdk"] Jul 14 15:21:34.637976 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.637958 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lznzq" Jul 14 15:21:34.638103 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.638086 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh"] Jul 14 15:21:34.638235 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.638221 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bf4cb94c7-wmnzm"] Jul 14 15:21:34.638358 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.638346 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jkhlz"] Jul 14 15:21:34.638468 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.638457 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-t4h7n"] Jul 14 15:21:34.638571 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.638559 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lznzq"] Jul 14 15:21:34.638682 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.638663 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xrzdx"] Jul 14 15:21:34.638876 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.638858 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-zxnts"] Jul 14 15:21:34.641932 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.641898 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 14 15:21:34.642093 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.642074 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xtkj9\"" Jul 14 15:21:34.642276 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.641947 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 14 15:21:34.642376 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.642336 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0509d07-fdb6-481b-b512-43cb8056858d-serving-cert\") pod \"service-ca-operator-7695fc6c8c-2zm28\" (UID: \"b0509d07-fdb6-481b-b512-43cb8056858d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2zm28" Jul 14 15:21:34.642615 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.642593 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b0509d07-fdb6-481b-b512-43cb8056858d-config\") pod \"service-ca-operator-7695fc6c8c-2zm28\" (UID: \"b0509d07-fdb6-481b-b512-43cb8056858d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2zm28" Jul 14 15:21:34.642699 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.642631 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvgm5\" (UniqueName: \"kubernetes.io/projected/99421093-ab46-4941-9ffe-f198f2d8b178-kube-api-access-pvgm5\") pod \"volume-data-source-validator-78495597f6-l52n9\" (UID: \"99421093-ab46-4941-9ffe-f198f2d8b178\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l52n9" Jul 14 15:21:34.642699 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.642675 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggwv4\" (UniqueName: \"kubernetes.io/projected/b0509d07-fdb6-481b-b512-43cb8056858d-kube-api-access-ggwv4\") pod \"service-ca-operator-7695fc6c8c-2zm28\" (UID: \"b0509d07-fdb6-481b-b512-43cb8056858d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2zm28" Jul 14 15:21:34.690682 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.690662 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:21:34.690781 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.690755 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:21:34.690827 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.690784 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:21:34.694505 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.694486 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 15:21:34.694602 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.694541 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 15:21:34.694680 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.694664 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vcjnm\"" Jul 14 15:21:34.694836 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.694824 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zdwzw\"" Jul 14 15:21:34.742998 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.742963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f022e7cf-42aa-4780-b3f0-6004f70ae107-cert\") pod \"ingress-canary-xrzdx\" (UID: \"f022e7cf-42aa-4780-b3f0-6004f70ae107\") " pod="openshift-ingress-canary/ingress-canary-xrzdx" Jul 14 15:21:34.743126 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf70d46f-232a-48f2-bb18-7050cfc08c69-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wtdm4\" (UID: \"cf70d46f-232a-48f2-bb18-7050cfc08c69\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdm4" Jul 14 15:21:34.743126 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743036 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls\") pod \"dns-default-lznzq\" (UID: \"d94ec422-3618-4b68-ae06-fc9232319c6d\") " pod="openshift-dns/dns-default-lznzq" Jul 14 15:21:34.743126 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743059 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cbb50f85-a13f-451d-a4bb-df7a9f212438-serving-cert\") pod \"insights-operator-68d9dd94cf-zxnts\" (UID: \"cbb50f85-a13f-451d-a4bb-df7a9f212438\") " pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" Jul 14 15:21:34.743126 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743103 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-bound-sa-token\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.743382 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743187 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7vn8\" (UniqueName: \"kubernetes.io/projected/f022e7cf-42aa-4780-b3f0-6004f70ae107-kube-api-access-t7vn8\") pod \"ingress-canary-xrzdx\" (UID: \"f022e7cf-42aa-4780-b3f0-6004f70ae107\") " pod="openshift-ingress-canary/ingress-canary-xrzdx" Jul 14 15:21:34.743382 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743227 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbb50f85-a13f-451d-a4bb-df7a9f212438-tmp\") pod \"insights-operator-68d9dd94cf-zxnts\" (UID: \"cbb50f85-a13f-451d-a4bb-df7a9f212438\") " pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" Jul 14 15:21:34.743382 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743257 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwmcc\" (UniqueName: \"kubernetes.io/projected/cf70d46f-232a-48f2-bb18-7050cfc08c69-kube-api-access-kwmcc\") pod \"cluster-samples-operator-65db858894-wtdm4\" (UID: \"cf70d46f-232a-48f2-bb18-7050cfc08c69\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdm4" Jul 14 15:21:34.743382 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0c30c26a-ed99-46f7-89d8-020e9ed3a7c8-serving-cert\") pod \"console-operator-7866497494-b9cq8\" (UID: \"0c30c26a-ed99-46f7-89d8-020e9ed3a7c8\") " pod="openshift-console-operator/console-operator-7866497494-b9cq8" Jul 14 15:21:34.743382 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743362 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0509d07-fdb6-481b-b512-43cb8056858d-serving-cert\") pod \"service-ca-operator-7695fc6c8c-2zm28\" (UID: \"b0509d07-fdb6-481b-b512-43cb8056858d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2zm28" Jul 14 15:21:34.743599 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d1490f8-b484-46ee-83d6-4bb7fae67fd8-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-jkhlz\" (UID: \"2d1490f8-b484-46ee-83d6-4bb7fae67fd8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jkhlz" Jul 14 15:21:34.743599 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743560 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d1490f8-b484-46ee-83d6-4bb7fae67fd8-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-jkhlz\" (UID: \"2d1490f8-b484-46ee-83d6-4bb7fae67fd8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jkhlz" Jul 14 15:21:34.743599 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743591 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-image-registry-private-configuration\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.743741 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b0509d07-fdb6-481b-b512-43cb8056858d-config\") pod \"service-ca-operator-7695fc6c8c-2zm28\" (UID: \"b0509d07-fdb6-481b-b512-43cb8056858d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2zm28" Jul 14 15:21:34.743741 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743663 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/02191573-3edc-4fe3-97af-d478beb7575c-nginx-conf\") pod \"networking-console-plugin-6cf856758c-4xn55\" (UID: \"02191573-3edc-4fe3-97af-d478beb7575c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-4xn55" Jul 14 15:21:34.743741 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743691 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-default-certificate\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:34.743741 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:34.743959 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743750 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.743959 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743768 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5gfj\" (UniqueName: \"kubernetes.io/projected/0c30c26a-ed99-46f7-89d8-020e9ed3a7c8-kube-api-access-c5gfj\") pod \"console-operator-7866497494-b9cq8\" (UID: \"0c30c26a-ed99-46f7-89d8-020e9ed3a7c8\") " pod="openshift-console-operator/console-operator-7866497494-b9cq8" Jul 14 15:21:34.743959 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggwv4\" (UniqueName: \"kubernetes.io/projected/b0509d07-fdb6-481b-b512-43cb8056858d-kube-api-access-ggwv4\") pod \"service-ca-operator-7695fc6c8c-2zm28\" (UID: \"b0509d07-fdb6-481b-b512-43cb8056858d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2zm28" Jul 14 15:21:34.743959 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743809 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-4xn55\" (UID: \"02191573-3edc-4fe3-97af-d478beb7575c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-4xn55" Jul 14 15:21:34.743959 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743826 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d94ec422-3618-4b68-ae06-fc9232319c6d-config-volume\") pod \"dns-default-lznzq\" (UID: \"d94ec422-3618-4b68-ae06-fc9232319c6d\") " pod="openshift-dns/dns-default-lznzq" Jul 14 15:21:34.743959 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743841 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d94ec422-3618-4b68-ae06-fc9232319c6d-tmp-dir\") pod \"dns-default-lznzq\" (UID: \"d94ec422-3618-4b68-ae06-fc9232319c6d\") " pod="openshift-dns/dns-default-lznzq" Jul 14 15:21:34.743959 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743858 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-96wbh\" (UID: \"dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh" Jul 14 15:21:34.743959 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743888 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0c30c26a-ed99-46f7-89d8-020e9ed3a7c8-config\") pod \"console-operator-7866497494-b9cq8\" (UID: \"0c30c26a-ed99-46f7-89d8-020e9ed3a7c8\") " pod="openshift-console-operator/console-operator-7866497494-b9cq8" Jul 14 15:21:34.743959 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743917 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kpsn\" (UniqueName: \"kubernetes.io/projected/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-kube-api-access-7kpsn\") pod \"cluster-monitoring-operator-5f7f494dd4-96wbh\" (UID: \"dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh" Jul 14 15:21:34.743959 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.743959 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc6q7\" (UniqueName: \"kubernetes.io/projected/cbb50f85-a13f-451d-a4bb-df7a9f212438-kube-api-access-dc6q7\") pod \"insights-operator-68d9dd94cf-zxnts\" (UID: \"cbb50f85-a13f-451d-a4bb-df7a9f212438\") " pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" Jul 14 15:21:34.744448 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.744009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-ca-trust-extracted\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.744448 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.744046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-96wbh\" (UID: \"dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh" Jul 14 15:21:34.744448 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.744100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbb50f85-a13f-451d-a4bb-df7a9f212438-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-zxnts\" (UID: \"cbb50f85-a13f-451d-a4bb-df7a9f212438\") " pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" Jul 14 15:21:34.744448 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.744126 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjv9h\" (UniqueName: \"kubernetes.io/projected/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-kube-api-access-gjv9h\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:34.744448 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.744149 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-installation-pull-secrets\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.744448 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.744198 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cbb50f85-a13f-451d-a4bb-df7a9f212438-snapshots\") pod \"insights-operator-68d9dd94cf-zxnts\" (UID: \"cbb50f85-a13f-451d-a4bb-df7a9f212438\") " pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" Jul 14 15:21:34.744448 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.744211 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b0509d07-fdb6-481b-b512-43cb8056858d-config\") pod \"service-ca-operator-7695fc6c8c-2zm28\" (UID: \"b0509d07-fdb6-481b-b512-43cb8056858d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2zm28" Jul 14 15:21:34.744448 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.744219 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmpb7\" (UniqueName: \"kubernetes.io/projected/d94ec422-3618-4b68-ae06-fc9232319c6d-kube-api-access-xmpb7\") pod \"dns-default-lznzq\" (UID: \"d94ec422-3618-4b68-ae06-fc9232319c6d\") " pod="openshift-dns/dns-default-lznzq" Jul 14 15:21:34.744448 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.744286 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h49hw\" (UniqueName: \"kubernetes.io/projected/75fe5b9c-5f1f-4448-b2b3-cbcf1066e23d-kube-api-access-h49hw\") pod \"network-check-source-869dcfcdcd-t4h7n\" (UID: \"75fe5b9c-5f1f-4448-b2b3-cbcf1066e23d\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-t4h7n" Jul 14 15:21:34.744448 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.744338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-stats-auth\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:34.744448 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.744377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvgm5\" (UniqueName: \"kubernetes.io/projected/99421093-ab46-4941-9ffe-f198f2d8b178-kube-api-access-pvgm5\") pod \"volume-data-source-validator-78495597f6-l52n9\" (UID: \"99421093-ab46-4941-9ffe-f198f2d8b178\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l52n9" Jul 14 15:21:34.744448 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.744404 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlq5d\" (UniqueName: \"kubernetes.io/projected/2d1490f8-b484-46ee-83d6-4bb7fae67fd8-kube-api-access-qlq5d\") pod \"kube-storage-version-migrator-operator-799cb7f958-jkhlz\" (UID: \"2d1490f8-b484-46ee-83d6-4bb7fae67fd8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jkhlz" Jul 14 15:21:34.744448 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.744429 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6n5w\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-kube-api-access-p6n5w\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.744988 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.744455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:34.744988 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.744493 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbb50f85-a13f-451d-a4bb-df7a9f212438-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-zxnts\" (UID: \"cbb50f85-a13f-451d-a4bb-df7a9f212438\") " pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" Jul 14 15:21:34.744988 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.744525 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-certificates\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.744988 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.744549 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-trusted-ca\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.744988 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.744579 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c30c26a-ed99-46f7-89d8-020e9ed3a7c8-trusted-ca\") pod \"console-operator-7866497494-b9cq8\" (UID: \"0c30c26a-ed99-46f7-89d8-020e9ed3a7c8\") " pod="openshift-console-operator/console-operator-7866497494-b9cq8" Jul 14 15:21:34.748416 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.748370 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0509d07-fdb6-481b-b512-43cb8056858d-serving-cert\") pod \"service-ca-operator-7695fc6c8c-2zm28\" (UID: \"b0509d07-fdb6-481b-b512-43cb8056858d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2zm28" Jul 14 15:21:34.756978 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.756954 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggwv4\" (UniqueName: \"kubernetes.io/projected/b0509d07-fdb6-481b-b512-43cb8056858d-kube-api-access-ggwv4\") pod \"service-ca-operator-7695fc6c8c-2zm28\" (UID: \"b0509d07-fdb6-481b-b512-43cb8056858d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2zm28" Jul 14 15:21:34.757343 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.757325 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvgm5\" (UniqueName: \"kubernetes.io/projected/99421093-ab46-4941-9ffe-f198f2d8b178-kube-api-access-pvgm5\") pod \"volume-data-source-validator-78495597f6-l52n9\" (UID: \"99421093-ab46-4941-9ffe-f198f2d8b178\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l52n9" Jul 14 15:21:34.845647 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.845608 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cbb50f85-a13f-451d-a4bb-df7a9f212438-snapshots\") pod \"insights-operator-68d9dd94cf-zxnts\" (UID: \"cbb50f85-a13f-451d-a4bb-df7a9f212438\") " pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" Jul 14 15:21:34.845814 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.845658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmpb7\" (UniqueName: \"kubernetes.io/projected/d94ec422-3618-4b68-ae06-fc9232319c6d-kube-api-access-xmpb7\") pod \"dns-default-lznzq\" (UID: \"d94ec422-3618-4b68-ae06-fc9232319c6d\") " pod="openshift-dns/dns-default-lznzq" Jul 14 15:21:34.845814 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.845788 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h49hw\" (UniqueName: \"kubernetes.io/projected/75fe5b9c-5f1f-4448-b2b3-cbcf1066e23d-kube-api-access-h49hw\") pod \"network-check-source-869dcfcdcd-t4h7n\" (UID: \"75fe5b9c-5f1f-4448-b2b3-cbcf1066e23d\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-t4h7n" Jul 14 15:21:34.845939 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.845827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-stats-auth\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:34.845939 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.845856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlq5d\" (UniqueName: \"kubernetes.io/projected/2d1490f8-b484-46ee-83d6-4bb7fae67fd8-kube-api-access-qlq5d\") pod \"kube-storage-version-migrator-operator-799cb7f958-jkhlz\" (UID: \"2d1490f8-b484-46ee-83d6-4bb7fae67fd8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jkhlz" Jul 14 15:21:34.845939 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.845884 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6n5w\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-kube-api-access-p6n5w\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.845939 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.845908 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:34.846143 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.845959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbb50f85-a13f-451d-a4bb-df7a9f212438-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-zxnts\" (UID: \"cbb50f85-a13f-451d-a4bb-df7a9f212438\") " pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" Jul 14 15:21:34.846143 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.846000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-certificates\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.846143 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.846033 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-trusted-ca\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.846143 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.846060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c30c26a-ed99-46f7-89d8-020e9ed3a7c8-trusted-ca\") pod \"console-operator-7866497494-b9cq8\" (UID: \"0c30c26a-ed99-46f7-89d8-020e9ed3a7c8\") " pod="openshift-console-operator/console-operator-7866497494-b9cq8" Jul 14 15:21:34.846143 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.846084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f022e7cf-42aa-4780-b3f0-6004f70ae107-cert\") pod \"ingress-canary-xrzdx\" (UID: \"f022e7cf-42aa-4780-b3f0-6004f70ae107\") " pod="openshift-ingress-canary/ingress-canary-xrzdx" Jul 14 15:21:34.846143 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.846110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf70d46f-232a-48f2-bb18-7050cfc08c69-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wtdm4\" (UID: \"cf70d46f-232a-48f2-bb18-7050cfc08c69\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdm4" Jul 14 15:21:34.846143 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.846130 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cbb50f85-a13f-451d-a4bb-df7a9f212438-snapshots\") pod \"insights-operator-68d9dd94cf-zxnts\" (UID: \"cbb50f85-a13f-451d-a4bb-df7a9f212438\") " pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" Jul 14 15:21:34.846143 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.846142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls\") pod \"dns-default-lznzq\" (UID: \"d94ec422-3618-4b68-ae06-fc9232319c6d\") " pod="openshift-dns/dns-default-lznzq" Jul 14 15:21:34.846528 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.846165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cbb50f85-a13f-451d-a4bb-df7a9f212438-serving-cert\") pod \"insights-operator-68d9dd94cf-zxnts\" (UID: \"cbb50f85-a13f-451d-a4bb-df7a9f212438\") " pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" Jul 14 15:21:34.846528 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.846215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-bound-sa-token\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.846528 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.846243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7vn8\" (UniqueName: \"kubernetes.io/projected/f022e7cf-42aa-4780-b3f0-6004f70ae107-kube-api-access-t7vn8\") pod \"ingress-canary-xrzdx\" (UID: \"f022e7cf-42aa-4780-b3f0-6004f70ae107\") " pod="openshift-ingress-canary/ingress-canary-xrzdx" Jul 14 15:21:34.846528 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.846271 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbb50f85-a13f-451d-a4bb-df7a9f212438-tmp\") pod \"insights-operator-68d9dd94cf-zxnts\" (UID: \"cbb50f85-a13f-451d-a4bb-df7a9f212438\") " pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" Jul 14 15:21:34.846528 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.846298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwmcc\" (UniqueName: \"kubernetes.io/projected/cf70d46f-232a-48f2-bb18-7050cfc08c69-kube-api-access-kwmcc\") pod \"cluster-samples-operator-65db858894-wtdm4\" (UID: \"cf70d46f-232a-48f2-bb18-7050cfc08c69\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdm4" Jul 14 15:21:34.846528 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.846323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0c30c26a-ed99-46f7-89d8-020e9ed3a7c8-serving-cert\") pod \"console-operator-7866497494-b9cq8\" (UID: \"0c30c26a-ed99-46f7-89d8-020e9ed3a7c8\") " pod="openshift-console-operator/console-operator-7866497494-b9cq8" Jul 14 15:21:34.846528 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.846381 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d1490f8-b484-46ee-83d6-4bb7fae67fd8-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-jkhlz\" (UID: \"2d1490f8-b484-46ee-83d6-4bb7fae67fd8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jkhlz" Jul 14 15:21:34.847068 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:34.846921 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 15:21:34.849898 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.847518 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbb50f85-a13f-451d-a4bb-df7a9f212438-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-zxnts\" (UID: \"cbb50f85-a13f-451d-a4bb-df7a9f212438\") " pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" Jul 14 15:21:34.849898 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:34.847702 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs podName:4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c nodeName:}" failed. No retries permitted until 2026-07-14 15:21:35.347676642 +0000 UTC m=+36.200059789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs") pod "router-default-6cd7778556-bmkdk" (UID: "4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c") : secret "router-metrics-certs-default" not found Jul 14 15:21:34.849898 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.847731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d1490f8-b484-46ee-83d6-4bb7fae67fd8-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-jkhlz\" (UID: \"2d1490f8-b484-46ee-83d6-4bb7fae67fd8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jkhlz" Jul 14 15:21:34.849898 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.847771 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-image-registry-private-configuration\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.849898 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.847814 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/02191573-3edc-4fe3-97af-d478beb7575c-nginx-conf\") pod \"networking-console-plugin-6cf856758c-4xn55\" (UID: \"02191573-3edc-4fe3-97af-d478beb7575c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-4xn55" Jul 14 15:21:34.849898 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.847846 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-default-certificate\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:34.849898 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.847877 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:34.849898 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.847915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.849898 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.847983 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5gfj\" (UniqueName: \"kubernetes.io/projected/0c30c26a-ed99-46f7-89d8-020e9ed3a7c8-kube-api-access-c5gfj\") pod \"console-operator-7866497494-b9cq8\" (UID: \"0c30c26a-ed99-46f7-89d8-020e9ed3a7c8\") " pod="openshift-console-operator/console-operator-7866497494-b9cq8" Jul 14 15:21:34.849898 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.848777 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-trusted-ca\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.849898 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.848805 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d1490f8-b484-46ee-83d6-4bb7fae67fd8-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-jkhlz\" (UID: \"2d1490f8-b484-46ee-83d6-4bb7fae67fd8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jkhlz" Jul 14 15:21:34.849898 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.849203 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-certificates\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.849898 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.849303 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-stats-auth\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:34.849898 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:34.849482 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 15:21:34.849898 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:34.849550 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf70d46f-232a-48f2-bb18-7050cfc08c69-samples-operator-tls podName:cf70d46f-232a-48f2-bb18-7050cfc08c69 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:35.349534736 +0000 UTC m=+36.201917894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cf70d46f-232a-48f2-bb18-7050cfc08c69-samples-operator-tls") pod "cluster-samples-operator-65db858894-wtdm4" (UID: "cf70d46f-232a-48f2-bb18-7050cfc08c69") : secret "samples-operator-tls" not found Jul 14 15:21:34.850729 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.849564 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbb50f85-a13f-451d-a4bb-df7a9f212438-tmp\") pod \"insights-operator-68d9dd94cf-zxnts\" (UID: \"cbb50f85-a13f-451d-a4bb-df7a9f212438\") " pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" Jul 14 15:21:34.850729 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.850075 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d1490f8-b484-46ee-83d6-4bb7fae67fd8-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-jkhlz\" (UID: \"2d1490f8-b484-46ee-83d6-4bb7fae67fd8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jkhlz" Jul 14 15:21:34.850729 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.850533 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/02191573-3edc-4fe3-97af-d478beb7575c-nginx-conf\") pod \"networking-console-plugin-6cf856758c-4xn55\" (UID: \"02191573-3edc-4fe3-97af-d478beb7575c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-4xn55" Jul 14 15:21:34.850729 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:34.850632 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle podName:4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c nodeName:}" failed. No retries permitted until 2026-07-14 15:21:35.350605966 +0000 UTC m=+36.202989129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle") pod "router-default-6cd7778556-bmkdk" (UID: "4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c") : configmap references non-existent config key: service-ca.crt Jul 14 15:21:34.850921 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:34.850757 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 15:21:34.850921 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:34.850772 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf4cb94c7-wmnzm: secret "image-registry-tls" not found Jul 14 15:21:34.850921 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:34.850819 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls podName:e4db9b13-7375-4fbc-bd78-ed7b42e75f2c nodeName:}" failed. No retries permitted until 2026-07-14 15:21:35.350804875 +0000 UTC m=+36.203188036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls") pod "image-registry-5bf4cb94c7-wmnzm" (UID: "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c") : secret "image-registry-tls" not found Jul 14 15:21:34.855196 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:34.851278 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 15:21:34.855196 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:34.851351 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls podName:d94ec422-3618-4b68-ae06-fc9232319c6d nodeName:}" failed. No retries permitted until 2026-07-14 15:21:35.351327505 +0000 UTC m=+36.203710650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls") pod "dns-default-lznzq" (UID: "d94ec422-3618-4b68-ae06-fc9232319c6d") : secret "dns-default-metrics-tls" not found Jul 14 15:21:34.855196 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.851373 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f022e7cf-42aa-4780-b3f0-6004f70ae107-cert\") pod \"ingress-canary-xrzdx\" (UID: \"f022e7cf-42aa-4780-b3f0-6004f70ae107\") " pod="openshift-ingress-canary/ingress-canary-xrzdx" Jul 14 15:21:34.855196 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.851512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-4xn55\" (UID: \"02191573-3edc-4fe3-97af-d478beb7575c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-4xn55" Jul 14 15:21:34.855196 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.851555 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d94ec422-3618-4b68-ae06-fc9232319c6d-config-volume\") pod \"dns-default-lznzq\" (UID: \"d94ec422-3618-4b68-ae06-fc9232319c6d\") " pod="openshift-dns/dns-default-lznzq" Jul 14 15:21:34.855196 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.851679 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d94ec422-3618-4b68-ae06-fc9232319c6d-tmp-dir\") pod \"dns-default-lznzq\" (UID: \"d94ec422-3618-4b68-ae06-fc9232319c6d\") " pod="openshift-dns/dns-default-lznzq" Jul 14 15:21:34.855196 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.851718 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-96wbh\" (UID: \"dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh" Jul 14 15:21:34.855196 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.851771 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0c30c26a-ed99-46f7-89d8-020e9ed3a7c8-config\") pod \"console-operator-7866497494-b9cq8\" (UID: \"0c30c26a-ed99-46f7-89d8-020e9ed3a7c8\") " pod="openshift-console-operator/console-operator-7866497494-b9cq8" Jul 14 15:21:34.855196 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.851777 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-default-certificate\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:34.855196 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.851803 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kpsn\" (UniqueName: \"kubernetes.io/projected/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-kube-api-access-7kpsn\") pod \"cluster-monitoring-operator-5f7f494dd4-96wbh\" (UID: \"dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh" Jul 14 15:21:34.855196 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.851835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc6q7\" (UniqueName: \"kubernetes.io/projected/cbb50f85-a13f-451d-a4bb-df7a9f212438-kube-api-access-dc6q7\") pod \"insights-operator-68d9dd94cf-zxnts\" (UID: \"cbb50f85-a13f-451d-a4bb-df7a9f212438\") " pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" Jul 14 15:21:34.855196 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.851860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-ca-trust-extracted\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.855196 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.851891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-96wbh\" (UID: \"dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh" Jul 14 15:21:34.855196 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.851934 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbb50f85-a13f-451d-a4bb-df7a9f212438-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-zxnts\" (UID: \"cbb50f85-a13f-451d-a4bb-df7a9f212438\") " pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" Jul 14 15:21:34.855196 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.852027 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjv9h\" (UniqueName: \"kubernetes.io/projected/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-kube-api-access-gjv9h\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:34.855196 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:34.852039 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 15:21:34.855905 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.852062 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-installation-pull-secrets\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.855905 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.852199 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d94ec422-3618-4b68-ae06-fc9232319c6d-tmp-dir\") pod \"dns-default-lznzq\" (UID: \"d94ec422-3618-4b68-ae06-fc9232319c6d\") " pod="openshift-dns/dns-default-lznzq" Jul 14 15:21:34.855905 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:34.852212 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert podName:02191573-3edc-4fe3-97af-d478beb7575c nodeName:}" failed. No retries permitted until 2026-07-14 15:21:35.352153076 +0000 UTC m=+36.204536236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-4xn55" (UID: "02191573-3edc-4fe3-97af-d478beb7575c") : secret "networking-console-plugin-cert" not found Jul 14 15:21:34.855905 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.852347 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c30c26a-ed99-46f7-89d8-020e9ed3a7c8-trusted-ca\") pod \"console-operator-7866497494-b9cq8\" (UID: \"0c30c26a-ed99-46f7-89d8-020e9ed3a7c8\") " pod="openshift-console-operator/console-operator-7866497494-b9cq8" Jul 14 15:21:34.855905 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.852614 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-ca-trust-extracted\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.855905 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:34.852661 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 15:21:34.855905 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.852686 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d94ec422-3618-4b68-ae06-fc9232319c6d-config-volume\") pod \"dns-default-lznzq\" (UID: \"d94ec422-3618-4b68-ae06-fc9232319c6d\") " pod="openshift-dns/dns-default-lznzq" Jul 14 15:21:34.855905 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:34.852714 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls podName:dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:35.35269319 +0000 UTC m=+36.205076336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-96wbh" (UID: "dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2") : secret "cluster-monitoring-operator-tls" not found Jul 14 15:21:34.855905 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.852984 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0c30c26a-ed99-46f7-89d8-020e9ed3a7c8-config\") pod \"console-operator-7866497494-b9cq8\" (UID: \"0c30c26a-ed99-46f7-89d8-020e9ed3a7c8\") " pod="openshift-console-operator/console-operator-7866497494-b9cq8" Jul 14 15:21:34.855905 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.853431 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0c30c26a-ed99-46f7-89d8-020e9ed3a7c8-serving-cert\") pod \"console-operator-7866497494-b9cq8\" (UID: \"0c30c26a-ed99-46f7-89d8-020e9ed3a7c8\") " pod="openshift-console-operator/console-operator-7866497494-b9cq8" Jul 14 15:21:34.855905 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.853874 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-96wbh\" (UID: \"dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh" Jul 14 15:21:34.855905 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.854411 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbb50f85-a13f-451d-a4bb-df7a9f212438-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-zxnts\" (UID: \"cbb50f85-a13f-451d-a4bb-df7a9f212438\") " pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" Jul 14 15:21:34.856412 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.856104 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-image-registry-private-configuration\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.860229 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.859138 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cbb50f85-a13f-451d-a4bb-df7a9f212438-serving-cert\") pod \"insights-operator-68d9dd94cf-zxnts\" (UID: \"cbb50f85-a13f-451d-a4bb-df7a9f212438\") " pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" Jul 14 15:21:34.860229 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.859232 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmpb7\" (UniqueName: \"kubernetes.io/projected/d94ec422-3618-4b68-ae06-fc9232319c6d-kube-api-access-xmpb7\") pod \"dns-default-lznzq\" (UID: \"d94ec422-3618-4b68-ae06-fc9232319c6d\") " pod="openshift-dns/dns-default-lznzq" Jul 14 15:21:34.860229 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.859483 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5gfj\" (UniqueName: \"kubernetes.io/projected/0c30c26a-ed99-46f7-89d8-020e9ed3a7c8-kube-api-access-c5gfj\") pod \"console-operator-7866497494-b9cq8\" (UID: \"0c30c26a-ed99-46f7-89d8-020e9ed3a7c8\") " pod="openshift-console-operator/console-operator-7866497494-b9cq8" Jul 14 15:21:34.860229 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.860202 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h49hw\" (UniqueName: \"kubernetes.io/projected/75fe5b9c-5f1f-4448-b2b3-cbcf1066e23d-kube-api-access-h49hw\") pod \"network-check-source-869dcfcdcd-t4h7n\" (UID: \"75fe5b9c-5f1f-4448-b2b3-cbcf1066e23d\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-t4h7n" Jul 14 15:21:34.860460 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.860279 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-installation-pull-secrets\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.860536 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.860500 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6n5w\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-kube-api-access-p6n5w\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.861273 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.861255 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2zm28" Jul 14 15:21:34.867121 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.867089 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l52n9" Jul 14 15:21:34.869762 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.869739 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-bound-sa-token\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:34.870871 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.870409 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7vn8\" (UniqueName: \"kubernetes.io/projected/f022e7cf-42aa-4780-b3f0-6004f70ae107-kube-api-access-t7vn8\") pod \"ingress-canary-xrzdx\" (UID: \"f022e7cf-42aa-4780-b3f0-6004f70ae107\") " pod="openshift-ingress-canary/ingress-canary-xrzdx" Jul 14 15:21:34.870871 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.870538 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlq5d\" (UniqueName: \"kubernetes.io/projected/2d1490f8-b484-46ee-83d6-4bb7fae67fd8-kube-api-access-qlq5d\") pod \"kube-storage-version-migrator-operator-799cb7f958-jkhlz\" (UID: \"2d1490f8-b484-46ee-83d6-4bb7fae67fd8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jkhlz" Jul 14 15:21:34.870871 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.870838 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwmcc\" (UniqueName: \"kubernetes.io/projected/cf70d46f-232a-48f2-bb18-7050cfc08c69-kube-api-access-kwmcc\") pod \"cluster-samples-operator-65db858894-wtdm4\" (UID: \"cf70d46f-232a-48f2-bb18-7050cfc08c69\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdm4" Jul 14 15:21:34.872807 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.872788 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kpsn\" (UniqueName: \"kubernetes.io/projected/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-kube-api-access-7kpsn\") pod \"cluster-monitoring-operator-5f7f494dd4-96wbh\" (UID: \"dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh" Jul 14 15:21:34.872883 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.872814 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjv9h\" (UniqueName: \"kubernetes.io/projected/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-kube-api-access-gjv9h\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:34.873212 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.873192 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc6q7\" (UniqueName: \"kubernetes.io/projected/cbb50f85-a13f-451d-a4bb-df7a9f212438-kube-api-access-dc6q7\") pod \"insights-operator-68d9dd94cf-zxnts\" (UID: \"cbb50f85-a13f-451d-a4bb-df7a9f212438\") " pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" Jul 14 15:21:34.913369 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.913338 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-b9cq8" Jul 14 15:21:34.921518 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.921496 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jkhlz" Jul 14 15:21:34.971086 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.971062 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" Jul 14 15:21:34.985068 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.985044 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xrzdx" Jul 14 15:21:34.990878 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:34.990860 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-t4h7n" Jul 14 15:21:35.356318 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:35.356278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:35.356493 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:35.356363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf70d46f-232a-48f2-bb18-7050cfc08c69-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wtdm4\" (UID: \"cf70d46f-232a-48f2-bb18-7050cfc08c69\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdm4" Jul 14 15:21:35.356493 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:35.356404 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls\") pod \"dns-default-lznzq\" (UID: \"d94ec422-3618-4b68-ae06-fc9232319c6d\") " pod="openshift-dns/dns-default-lznzq" Jul 14 15:21:35.356493 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:35.356458 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 15:21:35.356493 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:35.356466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:35.356493 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:35.356493 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 15:21:35.356686 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:35.356495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:35.356686 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:35.356531 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs podName:4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c nodeName:}" failed. No retries permitted until 2026-07-14 15:21:36.356511095 +0000 UTC m=+37.208894267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs") pod "router-default-6cd7778556-bmkdk" (UID: "4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c") : secret "router-metrics-certs-default" not found Jul 14 15:21:35.356686 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:35.356581 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 15:21:35.356686 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:35.356583 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 15:21:35.356686 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:35.356596 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf4cb94c7-wmnzm: secret "image-registry-tls" not found Jul 14 15:21:35.356686 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:35.356584 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf70d46f-232a-48f2-bb18-7050cfc08c69-samples-operator-tls podName:cf70d46f-232a-48f2-bb18-7050cfc08c69 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:36.35657256 +0000 UTC m=+37.208955703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cf70d46f-232a-48f2-bb18-7050cfc08c69-samples-operator-tls") pod "cluster-samples-operator-65db858894-wtdm4" (UID: "cf70d46f-232a-48f2-bb18-7050cfc08c69") : secret "samples-operator-tls" not found Jul 14 15:21:35.356686 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:35.356655 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle podName:4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c nodeName:}" failed. No retries permitted until 2026-07-14 15:21:36.356638258 +0000 UTC m=+37.209021403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle") pod "router-default-6cd7778556-bmkdk" (UID: "4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c") : configmap references non-existent config key: service-ca.crt Jul 14 15:21:35.356686 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:35.356686 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls podName:d94ec422-3618-4b68-ae06-fc9232319c6d nodeName:}" failed. No retries permitted until 2026-07-14 15:21:36.356673513 +0000 UTC m=+37.209056664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls") pod "dns-default-lznzq" (UID: "d94ec422-3618-4b68-ae06-fc9232319c6d") : secret "dns-default-metrics-tls" not found Jul 14 15:21:35.356978 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:35.356712 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls podName:e4db9b13-7375-4fbc-bd78-ed7b42e75f2c nodeName:}" failed. No retries permitted until 2026-07-14 15:21:36.356704175 +0000 UTC m=+37.209087318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls") pod "image-registry-5bf4cb94c7-wmnzm" (UID: "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c") : secret "image-registry-tls" not found Jul 14 15:21:35.356978 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:35.356746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-4xn55\" (UID: \"02191573-3edc-4fe3-97af-d478beb7575c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-4xn55" Jul 14 15:21:35.356978 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:35.356799 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-96wbh\" (UID: \"dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh" Jul 14 15:21:35.356978 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:35.356825 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 15:21:35.356978 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:35.356862 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert podName:02191573-3edc-4fe3-97af-d478beb7575c nodeName:}" failed. No retries permitted until 2026-07-14 15:21:36.356847299 +0000 UTC m=+37.209230441 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-4xn55" (UID: "02191573-3edc-4fe3-97af-d478beb7575c") : secret "networking-console-plugin-cert" not found Jul 14 15:21:35.356978 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:35.356886 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 15:21:35.356978 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:35.356918 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls podName:dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:36.356910174 +0000 UTC m=+37.209293317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-96wbh" (UID: "dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2") : secret "cluster-monitoring-operator-tls" not found Jul 14 15:21:36.392438 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.370145 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:36.392438 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.370743 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:36.392438 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.370787 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:36.392438 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.370818 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-4xn55\" (UID: \"02191573-3edc-4fe3-97af-d478beb7575c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-4xn55" Jul 14 15:21:36.392438 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.370863 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-96wbh\" (UID: \"dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh" Jul 14 15:21:36.392438 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.370909 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:36.392438 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.370952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf70d46f-232a-48f2-bb18-7050cfc08c69-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wtdm4\" (UID: \"cf70d46f-232a-48f2-bb18-7050cfc08c69\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdm4" Jul 14 15:21:36.392438 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.370983 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls\") pod \"dns-default-lznzq\" (UID: \"d94ec422-3618-4b68-ae06-fc9232319c6d\") " pod="openshift-dns/dns-default-lznzq" Jul 14 15:21:36.392438 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:36.371129 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 15:21:36.392438 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:36.371221 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls podName:d94ec422-3618-4b68-ae06-fc9232319c6d nodeName:}" failed. No retries permitted until 2026-07-14 15:21:38.371202277 +0000 UTC m=+39.223585424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls") pod "dns-default-lznzq" (UID: "d94ec422-3618-4b68-ae06-fc9232319c6d") : secret "dns-default-metrics-tls" not found Jul 14 15:21:36.392438 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:36.371381 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle podName:4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c nodeName:}" failed. No retries permitted until 2026-07-14 15:21:38.371366718 +0000 UTC m=+39.223749866 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle") pod "router-default-6cd7778556-bmkdk" (UID: "4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c") : configmap references non-existent config key: service-ca.crt Jul 14 15:21:36.392438 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:36.371455 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 15:21:36.392438 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:36.371466 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf4cb94c7-wmnzm: secret "image-registry-tls" not found Jul 14 15:21:36.392438 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:36.371498 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls podName:e4db9b13-7375-4fbc-bd78-ed7b42e75f2c nodeName:}" failed. No retries permitted until 2026-07-14 15:21:38.371487346 +0000 UTC m=+39.223870503 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls") pod "image-registry-5bf4cb94c7-wmnzm" (UID: "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c") : secret "image-registry-tls" not found Jul 14 15:21:36.392438 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:36.371552 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 15:21:36.392438 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:36.371583 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert podName:02191573-3edc-4fe3-97af-d478beb7575c nodeName:}" failed. No retries permitted until 2026-07-14 15:21:38.371573344 +0000 UTC m=+39.223956493 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-4xn55" (UID: "02191573-3edc-4fe3-97af-d478beb7575c") : secret "networking-console-plugin-cert" not found Jul 14 15:21:36.393885 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:36.371629 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 15:21:36.393885 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:36.371671 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls podName:dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:38.371655861 +0000 UTC m=+39.224039006 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-96wbh" (UID: "dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2") : secret "cluster-monitoring-operator-tls" not found Jul 14 15:21:36.393885 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:36.371715 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 15:21:36.393885 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:36.371743 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs podName:4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c nodeName:}" failed. No retries permitted until 2026-07-14 15:21:38.371734118 +0000 UTC m=+39.224117266 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs") pod "router-default-6cd7778556-bmkdk" (UID: "4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c") : secret "router-metrics-certs-default" not found Jul 14 15:21:36.393885 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:36.371789 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 15:21:36.393885 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:36.371816 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf70d46f-232a-48f2-bb18-7050cfc08c69-samples-operator-tls podName:cf70d46f-232a-48f2-bb18-7050cfc08c69 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:38.371808097 +0000 UTC m=+39.224191246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cf70d46f-232a-48f2-bb18-7050cfc08c69-samples-operator-tls") pod "cluster-samples-operator-65db858894-wtdm4" (UID: "cf70d46f-232a-48f2-bb18-7050cfc08c69") : secret "samples-operator-tls" not found Jul 14 15:21:36.458416 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.457431 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:36.527836 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.527805 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l52n9"] Jul 14 15:21:36.531258 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:21:36.531225 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99421093_ab46_4941_9ffe_f198f2d8b178.slice/crio-4e6bc29cb8e88e916a18f03fe3903c734b73d595e2e144a5b3a51aa86f133a82 WatchSource:0}: Error finding container 4e6bc29cb8e88e916a18f03fe3903c734b73d595e2e144a5b3a51aa86f133a82: Status 404 returned error can't find the container with id 4e6bc29cb8e88e916a18f03fe3903c734b73d595e2e144a5b3a51aa86f133a82 Jul 14 15:21:36.534743 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.534682 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-zxnts"] Jul 14 15:21:36.539154 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:21:36.539129 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb50f85_a13f_451d_a4bb_df7a9f212438.slice/crio-e06952b3efa88a725e870aa9ea674334ad2199f1ba929a115761c738f39b646e WatchSource:0}: Error finding container e06952b3efa88a725e870aa9ea674334ad2199f1ba929a115761c738f39b646e: Status 404 returned error can't find the container with id e06952b3efa88a725e870aa9ea674334ad2199f1ba929a115761c738f39b646e Jul 14 15:21:36.539915 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.539895 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-b9cq8"] Jul 14 15:21:36.544358 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.544115 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jkhlz"] Jul 14 15:21:36.546635 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.545892 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2zm28"] Jul 14 15:21:36.549165 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:21:36.548583 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0509d07_fdb6_481b_b512_43cb8056858d.slice/crio-519fbec0cddff55768134a526fdf5c7f1b0816cf11ef297f8a591649d0808c3d WatchSource:0}: Error finding container 519fbec0cddff55768134a526fdf5c7f1b0816cf11ef297f8a591649d0808c3d: Status 404 returned error can't find the container with id 519fbec0cddff55768134a526fdf5c7f1b0816cf11ef297f8a591649d0808c3d Jul 14 15:21:36.549943 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:21:36.549920 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1490f8_b484_46ee_83d6_4bb7fae67fd8.slice/crio-8339f56fbb0f90a40ac5aed6f3a35cd2c1945aa3b84f20d9e22deedf60e2476a WatchSource:0}: Error finding container 8339f56fbb0f90a40ac5aed6f3a35cd2c1945aa3b84f20d9e22deedf60e2476a: Status 404 returned error can't find the container with id 8339f56fbb0f90a40ac5aed6f3a35cd2c1945aa3b84f20d9e22deedf60e2476a Jul 14 15:21:36.553722 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.553692 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xrzdx"] Jul 14 15:21:36.556913 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.556890 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-t4h7n"] Jul 14 15:21:36.557278 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:21:36.557256 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf022e7cf_42aa_4780_b3f0_6004f70ae107.slice/crio-817f2a865eae1aa42360d7c5bd716df58ba5f39c8caa2a00a61ea35358c304a1 WatchSource:0}: Error finding container 817f2a865eae1aa42360d7c5bd716df58ba5f39c8caa2a00a61ea35358c304a1: Status 404 returned error can't find the container with id 817f2a865eae1aa42360d7c5bd716df58ba5f39c8caa2a00a61ea35358c304a1 Jul 14 15:21:36.560331 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:21:36.560310 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75fe5b9c_5f1f_4448_b2b3_cbcf1066e23d.slice/crio-16563074b993d936bb1ff7a470ef6cae7c59100e3061a7e863e5660708216ede WatchSource:0}: Error finding container 16563074b993d936bb1ff7a470ef6cae7c59100e3061a7e863e5660708216ede: Status 404 returned error can't find the container with id 16563074b993d936bb1ff7a470ef6cae7c59100e3061a7e863e5660708216ede Jul 14 15:21:36.903937 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.903900 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l52n9" event={"ID":"99421093-ab46-4941-9ffe-f198f2d8b178","Type":"ContainerStarted","Data":"4e6bc29cb8e88e916a18f03fe3903c734b73d595e2e144a5b3a51aa86f133a82"} Jul 14 15:21:36.905026 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.904997 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-b9cq8" event={"ID":"0c30c26a-ed99-46f7-89d8-020e9ed3a7c8","Type":"ContainerStarted","Data":"4e8fc6f7f59afff14656c73b779f9b184b77ba4c957560a920c33f4293ae494c"} Jul 14 15:21:36.907659 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.907635 2579 generic.go:356] "Generic (PLEG): container finished" podID="5062eb6e-d839-4164-aad5-057e03ade017" containerID="3bcc054daa3b8baeb4f64165e8345b4d01af346411b928fa8afad412138942be" exitCode=0 Jul 14 15:21:36.907745 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.907699 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvkjc" event={"ID":"5062eb6e-d839-4164-aad5-057e03ade017","Type":"ContainerDied","Data":"3bcc054daa3b8baeb4f64165e8345b4d01af346411b928fa8afad412138942be"} Jul 14 15:21:36.908826 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.908793 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xrzdx" event={"ID":"f022e7cf-42aa-4780-b3f0-6004f70ae107","Type":"ContainerStarted","Data":"817f2a865eae1aa42360d7c5bd716df58ba5f39c8caa2a00a61ea35358c304a1"} Jul 14 15:21:36.910008 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.909984 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-t4h7n" event={"ID":"75fe5b9c-5f1f-4448-b2b3-cbcf1066e23d","Type":"ContainerStarted","Data":"16563074b993d936bb1ff7a470ef6cae7c59100e3061a7e863e5660708216ede"} Jul 14 15:21:36.911103 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.911077 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jkhlz" event={"ID":"2d1490f8-b484-46ee-83d6-4bb7fae67fd8","Type":"ContainerStarted","Data":"8339f56fbb0f90a40ac5aed6f3a35cd2c1945aa3b84f20d9e22deedf60e2476a"} Jul 14 15:21:36.912184 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.912149 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" event={"ID":"cbb50f85-a13f-451d-a4bb-df7a9f212438","Type":"ContainerStarted","Data":"e06952b3efa88a725e870aa9ea674334ad2199f1ba929a115761c738f39b646e"} Jul 14 15:21:36.913269 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:36.913251 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2zm28" event={"ID":"b0509d07-fdb6-481b-b512-43cb8056858d","Type":"ContainerStarted","Data":"519fbec0cddff55768134a526fdf5c7f1b0816cf11ef297f8a591649d0808c3d"} Jul 14 15:21:37.927684 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:37.927552 2579 generic.go:356] "Generic (PLEG): container finished" podID="5062eb6e-d839-4164-aad5-057e03ade017" containerID="8b19e9835b6e3e7e3970a4716bcbe63da66ce530150a20f3b6f7dc8a79eee397" exitCode=0 Jul 14 15:21:37.927684 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:37.927612 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvkjc" event={"ID":"5062eb6e-d839-4164-aad5-057e03ade017","Type":"ContainerDied","Data":"8b19e9835b6e3e7e3970a4716bcbe63da66ce530150a20f3b6f7dc8a79eee397"} Jul 14 15:21:38.394409 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:38.394374 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf70d46f-232a-48f2-bb18-7050cfc08c69-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wtdm4\" (UID: \"cf70d46f-232a-48f2-bb18-7050cfc08c69\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdm4" Jul 14 15:21:38.394594 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:38.394418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls\") pod \"dns-default-lznzq\" (UID: \"d94ec422-3618-4b68-ae06-fc9232319c6d\") " pod="openshift-dns/dns-default-lznzq" Jul 14 15:21:38.394594 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:38.394480 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:38.394594 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:38.394513 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:38.394594 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:38.394533 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 15:21:38.394594 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:38.394545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-4xn55\" (UID: \"02191573-3edc-4fe3-97af-d478beb7575c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-4xn55" Jul 14 15:21:38.394594 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:38.394598 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf70d46f-232a-48f2-bb18-7050cfc08c69-samples-operator-tls podName:cf70d46f-232a-48f2-bb18-7050cfc08c69 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:42.39457944 +0000 UTC m=+43.246962599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cf70d46f-232a-48f2-bb18-7050cfc08c69-samples-operator-tls") pod "cluster-samples-operator-65db858894-wtdm4" (UID: "cf70d46f-232a-48f2-bb18-7050cfc08c69") : secret "samples-operator-tls" not found Jul 14 15:21:38.394824 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:38.394632 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 15:21:38.394824 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:38.394668 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert podName:02191573-3edc-4fe3-97af-d478beb7575c nodeName:}" failed. No retries permitted until 2026-07-14 15:21:42.394656916 +0000 UTC m=+43.247040061 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-4xn55" (UID: "02191573-3edc-4fe3-97af-d478beb7575c") : secret "networking-console-plugin-cert" not found Jul 14 15:21:38.394824 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:38.394669 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 15:21:38.394824 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:38.394693 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf4cb94c7-wmnzm: secret "image-registry-tls" not found Jul 14 15:21:38.394824 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:38.394683 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle podName:4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c nodeName:}" failed. No retries permitted until 2026-07-14 15:21:42.394674734 +0000 UTC m=+43.247057879 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle") pod "router-default-6cd7778556-bmkdk" (UID: "4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c") : configmap references non-existent config key: service-ca.crt Jul 14 15:21:38.394824 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:38.394742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-96wbh\" (UID: \"dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh" Jul 14 15:21:38.394824 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:38.394674 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 15:21:38.394824 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:38.394810 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls podName:d94ec422-3618-4b68-ae06-fc9232319c6d nodeName:}" failed. No retries permitted until 2026-07-14 15:21:42.394794257 +0000 UTC m=+43.247177402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls") pod "dns-default-lznzq" (UID: "d94ec422-3618-4b68-ae06-fc9232319c6d") : secret "dns-default-metrics-tls" not found Jul 14 15:21:38.394824 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:38.394815 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 15:21:38.394824 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:38.394825 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:38.395164 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:38.394861 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls podName:dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:42.394842711 +0000 UTC m=+43.247225855 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-96wbh" (UID: "dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2") : secret "cluster-monitoring-operator-tls" not found Jul 14 15:21:38.395164 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:38.394897 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls podName:e4db9b13-7375-4fbc-bd78-ed7b42e75f2c nodeName:}" failed. No retries permitted until 2026-07-14 15:21:42.394883472 +0000 UTC m=+43.247266618 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls") pod "image-registry-5bf4cb94c7-wmnzm" (UID: "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c") : secret "image-registry-tls" not found Jul 14 15:21:38.395164 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:38.394946 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 15:21:38.395164 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:38.395016 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs podName:4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c nodeName:}" failed. No retries permitted until 2026-07-14 15:21:42.394997931 +0000 UTC m=+43.247381080 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs") pod "router-default-6cd7778556-bmkdk" (UID: "4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c") : secret "router-metrics-certs-default" not found Jul 14 15:21:38.923012 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:38.922976 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:38.934916 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:38.934873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvkjc" event={"ID":"5062eb6e-d839-4164-aad5-057e03ade017","Type":"ContainerStarted","Data":"1abbe6c67e27565e7dec080f888ee7e949efdafb859e25e364cb60ed4c75602b"} Jul 14 15:21:38.970518 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:38.970470 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kvkjc" podStartSLOduration=8.02177321 podStartE2EDuration="39.970454052s" totalImagesPullingTime="31.948680842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:20:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:38.970128771 +0000 UTC m=+39.822511939" watchObservedRunningTime="2026-07-14 15:21:38.970454052 +0000 UTC m=+39.822837216" Jul 14 15:21:39.123126 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:39.123084 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:39.225465 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:39.225400 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:39.323873 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:39.323832 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:39.435281 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:39.435231 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:39.549810 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:39.549723 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:39.650057 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:39.649981 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:40.546970 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:40.546748 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:40.644990 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:40.644950 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:42.430430 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:42.430386 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:42.431275 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:42.431249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:42.431406 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:42.431288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:42.431548 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:42.431312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-4xn55\" (UID: \"02191573-3edc-4fe3-97af-d478beb7575c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-4xn55" Jul 14 15:21:42.436064 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:42.431773 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-96wbh\" (UID: \"dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh" Jul 14 15:21:42.436064 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:42.431855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:42.436064 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:42.431779 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 15:21:42.436064 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:42.431882 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle podName:4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c nodeName:}" failed. No retries permitted until 2026-07-14 15:21:50.431862021 +0000 UTC m=+51.284245169 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle") pod "router-default-6cd7778556-bmkdk" (UID: "4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c") : configmap references non-existent config key: service-ca.crt Jul 14 15:21:42.436064 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:42.431946 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 15:21:42.436064 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:42.431974 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf70d46f-232a-48f2-bb18-7050cfc08c69-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wtdm4\" (UID: \"cf70d46f-232a-48f2-bb18-7050cfc08c69\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdm4" Jul 14 15:21:42.436064 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:42.431901 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf4cb94c7-wmnzm: secret "image-registry-tls" not found Jul 14 15:21:42.436064 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:42.432020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls\") pod \"dns-default-lznzq\" (UID: \"d94ec422-3618-4b68-ae06-fc9232319c6d\") " pod="openshift-dns/dns-default-lznzq" Jul 14 15:21:42.436064 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:42.432037 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 15:21:42.436064 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:42.431831 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 15:21:42.436064 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:42.432068 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls podName:dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:50.432048169 +0000 UTC m=+51.284431313 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-96wbh" (UID: "dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2") : secret "cluster-monitoring-operator-tls" not found Jul 14 15:21:42.436064 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:42.432103 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs podName:4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c nodeName:}" failed. No retries permitted until 2026-07-14 15:21:50.43208457 +0000 UTC m=+51.284467715 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs") pod "router-default-6cd7778556-bmkdk" (UID: "4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c") : secret "router-metrics-certs-default" not found Jul 14 15:21:42.436064 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:42.432127 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls podName:e4db9b13-7375-4fbc-bd78-ed7b42e75f2c nodeName:}" failed. No retries permitted until 2026-07-14 15:21:50.432115149 +0000 UTC m=+51.284498292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls") pod "image-registry-5bf4cb94c7-wmnzm" (UID: "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c") : secret "image-registry-tls" not found Jul 14 15:21:42.436064 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:42.432129 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 15:21:42.436064 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:42.432153 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert podName:02191573-3edc-4fe3-97af-d478beb7575c nodeName:}" failed. No retries permitted until 2026-07-14 15:21:50.432136494 +0000 UTC m=+51.284519645 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-4xn55" (UID: "02191573-3edc-4fe3-97af-d478beb7575c") : secret "networking-console-plugin-cert" not found Jul 14 15:21:42.436064 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:42.432192 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 15:21:42.436981 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:42.432193 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls podName:d94ec422-3618-4b68-ae06-fc9232319c6d nodeName:}" failed. No retries permitted until 2026-07-14 15:21:50.432162425 +0000 UTC m=+51.284545569 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls") pod "dns-default-lznzq" (UID: "d94ec422-3618-4b68-ae06-fc9232319c6d") : secret "dns-default-metrics-tls" not found Jul 14 15:21:42.436981 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:42.432272 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf70d46f-232a-48f2-bb18-7050cfc08c69-samples-operator-tls podName:cf70d46f-232a-48f2-bb18-7050cfc08c69 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:50.432251107 +0000 UTC m=+51.284634252 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cf70d46f-232a-48f2-bb18-7050cfc08c69-samples-operator-tls") pod "cluster-samples-operator-65db858894-wtdm4" (UID: "cf70d46f-232a-48f2-bb18-7050cfc08c69") : secret "samples-operator-tls" not found Jul 14 15:21:42.618562 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:42.618521 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:42.673432 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:42.673331 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:42.948375 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:42.947841 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2zm28" event={"ID":"b0509d07-fdb6-481b-b512-43cb8056858d","Type":"ContainerStarted","Data":"7bf63d4e12b1bafa0e0ecacfc28d4aef53e09911e8956133f4684f868462e4dd"} Jul 14 15:21:42.950308 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:42.950276 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l52n9" event={"ID":"99421093-ab46-4941-9ffe-f198f2d8b178","Type":"ContainerStarted","Data":"87558cbc5bd700651ab9c5ef2466476d94a5f61a3a1e0a9673e655e05ba8991d"} Jul 14 15:21:42.952319 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:42.951936 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-b9cq8" event={"ID":"0c30c26a-ed99-46f7-89d8-020e9ed3a7c8","Type":"ContainerStarted","Data":"9a96d95caf2de914af90fbbd1812bd080351921fcf8cf75c6029a8a151dee316"} Jul 14 15:21:42.952319 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:42.952049 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-b9cq8" Jul 14 15:21:42.953443 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:42.953417 2579 patch_prober.go:28] interesting pod/console-operator-7866497494-b9cq8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.8:8443/readyz\": dial tcp 10.132.0.8:8443: connect: connection refused" start-of-body= Jul 14 15:21:42.953550 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:42.953481 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7866497494-b9cq8" podUID="0c30c26a-ed99-46f7-89d8-020e9ed3a7c8" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.8:8443/readyz\": dial tcp 10.132.0.8:8443: connect: connection refused" Jul 14 15:21:42.953927 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:42.953909 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jkhlz" event={"ID":"2d1490f8-b484-46ee-83d6-4bb7fae67fd8","Type":"ContainerStarted","Data":"2b7aa34818571c6f1bb0a865a630dd8aac6e1438d2ae30785608f612859419e2"} Jul 14 15:21:42.965856 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:42.965808 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2zm28" podStartSLOduration=23.344752458 podStartE2EDuration="28.965792984s" totalImagesPullingTime="5.621040526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:42.964762582 +0000 UTC m=+43.817145751" watchObservedRunningTime="2026-07-14 15:21:42.965792984 +0000 UTC m=+43.818176150" Jul 14 15:21:42.987122 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:42.985351 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l52n9" podStartSLOduration=23.352392614 podStartE2EDuration="28.985332705s" totalImagesPullingTime="5.632940091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:42.984057891 +0000 UTC m=+43.836441056" watchObservedRunningTime="2026-07-14 15:21:42.985332705 +0000 UTC m=+43.837715871" Jul 14 15:21:43.029315 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.029259 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jkhlz" podStartSLOduration=23.414857843 podStartE2EDuration="29.029236736s" totalImagesPullingTime="5.614378893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:43.007959109 +0000 UTC m=+43.860342275" watchObservedRunningTime="2026-07-14 15:21:43.029236736 +0000 UTC m=+43.881619903" Jul 14 15:21:43.147051 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.147015 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:43.247296 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.247201 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:43.397814 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.397753 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-b9cq8" podStartSLOduration=23.777752555 podStartE2EDuration="29.397725635s" totalImagesPullingTime="5.61997308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:43.030558355 +0000 UTC m=+43.882941521" watchObservedRunningTime="2026-07-14 15:21:43.397725635 +0000 UTC m=+44.250108801" Jul 14 15:21:43.398988 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.398961 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-jhjxc"] Jul 14 15:21:43.402307 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.402279 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:43.417041 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.416882 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-jhjxc"] Jul 14 15:21:43.417041 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.417000 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-jhjxc" Jul 14 15:21:43.423484 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.420381 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ggpvr\"" Jul 14 15:21:43.423484 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.420685 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 14 15:21:43.423484 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.420905 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 14 15:21:43.445215 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.445160 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8dvj\" (UniqueName: \"kubernetes.io/projected/cb4ec4df-6611-41c7-b1aa-f32a58b3a251-kube-api-access-p8dvj\") pod \"migrator-7f59b54f6c-jhjxc\" (UID: \"cb4ec4df-6611-41c7-b1aa-f32a58b3a251\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-jhjxc" Jul 14 15:21:43.465275 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.465245 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:43.546945 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.546854 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8dvj\" (UniqueName: \"kubernetes.io/projected/cb4ec4df-6611-41c7-b1aa-f32a58b3a251-kube-api-access-p8dvj\") pod \"migrator-7f59b54f6c-jhjxc\" (UID: \"cb4ec4df-6611-41c7-b1aa-f32a58b3a251\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-jhjxc" Jul 14 15:21:43.558421 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.558392 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8dvj\" (UniqueName: \"kubernetes.io/projected/cb4ec4df-6611-41c7-b1aa-f32a58b3a251-kube-api-access-p8dvj\") pod \"migrator-7f59b54f6c-jhjxc\" (UID: \"cb4ec4df-6611-41c7-b1aa-f32a58b3a251\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-jhjxc" Jul 14 15:21:43.566251 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.566138 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:43.666101 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.666063 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:43.731381 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.731343 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-jhjxc" Jul 14 15:21:43.958379 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.958353 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/0.log" Jul 14 15:21:43.958548 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.958390 2579 generic.go:356] "Generic (PLEG): container finished" podID="0c30c26a-ed99-46f7-89d8-020e9ed3a7c8" containerID="9a96d95caf2de914af90fbbd1812bd080351921fcf8cf75c6029a8a151dee316" exitCode=255 Jul 14 15:21:43.958548 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.958464 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-b9cq8" event={"ID":"0c30c26a-ed99-46f7-89d8-020e9ed3a7c8","Type":"ContainerDied","Data":"9a96d95caf2de914af90fbbd1812bd080351921fcf8cf75c6029a8a151dee316"} Jul 14 15:21:43.958707 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:43.958692 2579 scope.go:117] "RemoveContainer" containerID="9a96d95caf2de914af90fbbd1812bd080351921fcf8cf75c6029a8a151dee316" Jul 14 15:21:44.514682 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:44.514508 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-jhjxc"] Jul 14 15:21:44.568213 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:44.568137 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:44.669397 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:44.669363 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:44.914082 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:44.913993 2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-b9cq8" Jul 14 15:21:44.964383 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:44.964359 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 15:21:44.964858 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:44.964839 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/0.log" Jul 14 15:21:44.964929 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:44.964885 2579 generic.go:356] "Generic (PLEG): container finished" podID="0c30c26a-ed99-46f7-89d8-020e9ed3a7c8" containerID="11e1b47fefc2f3369a8c4bc6b9284cc5fc21160f25956849a867e3adca597484" exitCode=255 Jul 14 15:21:44.965002 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:44.964977 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-b9cq8" event={"ID":"0c30c26a-ed99-46f7-89d8-020e9ed3a7c8","Type":"ContainerDied","Data":"11e1b47fefc2f3369a8c4bc6b9284cc5fc21160f25956849a867e3adca597484"} Jul 14 15:21:44.965193 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:44.965057 2579 scope.go:117] "RemoveContainer" containerID="9a96d95caf2de914af90fbbd1812bd080351921fcf8cf75c6029a8a151dee316" Jul 14 15:21:44.965314 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:44.965275 2579 scope.go:117] "RemoveContainer" containerID="11e1b47fefc2f3369a8c4bc6b9284cc5fc21160f25956849a867e3adca597484" Jul 14 15:21:44.965482 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:44.965462 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-b9cq8_openshift-console-operator(0c30c26a-ed99-46f7-89d8-020e9ed3a7c8)\"" pod="openshift-console-operator/console-operator-7866497494-b9cq8" podUID="0c30c26a-ed99-46f7-89d8-020e9ed3a7c8" Jul 14 15:21:44.966915 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:44.966895 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xrzdx" event={"ID":"f022e7cf-42aa-4780-b3f0-6004f70ae107","Type":"ContainerStarted","Data":"883ff5cd19deb3657a3a29a488e5570b0edf210e63bcad29ccf25c95ac81a86b"} Jul 14 15:21:44.968585 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:44.968558 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-t4h7n" event={"ID":"75fe5b9c-5f1f-4448-b2b3-cbcf1066e23d","Type":"ContainerStarted","Data":"57263f57f9095a3083d59799a95c5ace75e6ecc1415b6a071df7076979e49424"} Jul 14 15:21:44.970138 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:44.970116 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" event={"ID":"cbb50f85-a13f-451d-a4bb-df7a9f212438","Type":"ContainerStarted","Data":"c7db711a12f1cec341845aa6429f0f92d8b39e5a26bf3dd046d959bb4acad0f0"} Jul 14 15:21:44.971295 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:44.971275 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-jhjxc" event={"ID":"cb4ec4df-6611-41c7-b1aa-f32a58b3a251","Type":"ContainerStarted","Data":"d1bab4cdc0265882c1d07c51d938533c2454f3ce8b1840fcd7beadb8cbecac30"} Jul 14 15:21:45.001205 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:45.001147 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" podStartSLOduration=23.179308052 podStartE2EDuration="31.001134075s" totalImagesPullingTime="7.821826023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:45.000944502 +0000 UTC m=+45.853327704" watchObservedRunningTime="2026-07-14 15:21:45.001134075 +0000 UTC m=+45.853517242" Jul 14 15:21:45.018347 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:45.018286 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xrzdx" podStartSLOduration=3.206943375 podStartE2EDuration="11.018266549s" totalImagesPullingTime="7.811323174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:45.017468279 +0000 UTC m=+45.869851445" watchObservedRunningTime="2026-07-14 15:21:45.018266549 +0000 UTC m=+45.870649715" Jul 14 15:21:45.022512 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:45.022481 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:45.037376 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:45.036401 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-t4h7n" podStartSLOduration=23.273964333 podStartE2EDuration="31.036385776s" totalImagesPullingTime="7.762421443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:45.036118208 +0000 UTC m=+45.888501374" watchObservedRunningTime="2026-07-14 15:21:45.036385776 +0000 UTC m=+45.888768941" Jul 14 15:21:45.123246 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:45.123207 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:45.458315 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:45.458262 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:45.510313 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:45.510272 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:45.718979 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:45.718898 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:45.818532 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:45.818497 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:45.977863 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:45.977787 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 15:21:45.978313 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:45.978289 2579 scope.go:117] "RemoveContainer" containerID="11e1b47fefc2f3369a8c4bc6b9284cc5fc21160f25956849a867e3adca597484" Jul 14 15:21:45.978511 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:45.978484 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-b9cq8_openshift-console-operator(0c30c26a-ed99-46f7-89d8-020e9ed3a7c8)\"" pod="openshift-console-operator/console-operator-7866497494-b9cq8" podUID="0c30c26a-ed99-46f7-89d8-020e9ed3a7c8" Jul 14 15:21:46.070136 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.070098 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xmsfd"] Jul 14 15:21:46.081450 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.081423 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:46.083600 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.083573 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xmsfd"] Jul 14 15:21:46.084773 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.084752 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 14 15:21:46.084773 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.084759 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 14 15:21:46.085159 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.085139 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bllnj\"" Jul 14 15:21:46.105420 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.105395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:46.173627 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.173591 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/28db3bec-425f-408b-b575-11bbe25511b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:46.173744 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.173660 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/28db3bec-425f-408b-b575-11bbe25511b7-crio-socket\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:46.173744 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.173685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/28db3bec-425f-408b-b575-11bbe25511b7-data-volume\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:46.173744 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.173733 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/28db3bec-425f-408b-b575-11bbe25511b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:46.173905 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.173867 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn22x\" (UniqueName: \"kubernetes.io/projected/28db3bec-425f-408b-b575-11bbe25511b7-kube-api-access-bn22x\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:46.274980 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.274906 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/28db3bec-425f-408b-b575-11bbe25511b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:46.275118 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.274983 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/28db3bec-425f-408b-b575-11bbe25511b7-crio-socket\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:46.275118 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.275010 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/28db3bec-425f-408b-b575-11bbe25511b7-data-volume\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:46.275118 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:46.275035 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 15:21:46.275118 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.275061 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/28db3bec-425f-408b-b575-11bbe25511b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:46.275339 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:46.275121 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28db3bec-425f-408b-b575-11bbe25511b7-kube-rbac-proxy-cm podName:28db3bec-425f-408b-b575-11bbe25511b7 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:46.775096009 +0000 UTC m=+47.627479151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/28db3bec-425f-408b-b575-11bbe25511b7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmsfd" (UID: "28db3bec-425f-408b-b575-11bbe25511b7") : configmap "kube-rbac-proxy" not found Jul 14 15:21:46.275339 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:46.275196 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 15:21:46.275339 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:46.275249 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28db3bec-425f-408b-b575-11bbe25511b7-insights-runtime-extractor-tls podName:28db3bec-425f-408b-b575-11bbe25511b7 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:46.77523382 +0000 UTC m=+47.627616967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/28db3bec-425f-408b-b575-11bbe25511b7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmsfd" (UID: "28db3bec-425f-408b-b575-11bbe25511b7") : secret "insights-runtime-extractor-tls" not found Jul 14 15:21:46.275339 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.275242 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/28db3bec-425f-408b-b575-11bbe25511b7-crio-socket\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:46.275339 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.275242 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn22x\" (UniqueName: \"kubernetes.io/projected/28db3bec-425f-408b-b575-11bbe25511b7-kube-api-access-bn22x\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:46.275545 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.275373 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/28db3bec-425f-408b-b575-11bbe25511b7-data-volume\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:46.284600 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.284568 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn22x\" (UniqueName: \"kubernetes.io/projected/28db3bec-425f-408b-b575-11bbe25511b7-kube-api-access-bn22x\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:46.780944 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.780911 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/28db3bec-425f-408b-b575-11bbe25511b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:46.781337 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.780977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/28db3bec-425f-408b-b575-11bbe25511b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:46.781337 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:46.781037 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 15:21:46.781337 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:46.781073 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 15:21:46.781337 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:46.781106 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28db3bec-425f-408b-b575-11bbe25511b7-kube-rbac-proxy-cm podName:28db3bec-425f-408b-b575-11bbe25511b7 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:47.781090427 +0000 UTC m=+48.633473569 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/28db3bec-425f-408b-b575-11bbe25511b7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmsfd" (UID: "28db3bec-425f-408b-b575-11bbe25511b7") : configmap "kube-rbac-proxy" not found Jul 14 15:21:46.781337 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:46.781120 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28db3bec-425f-408b-b575-11bbe25511b7-insights-runtime-extractor-tls podName:28db3bec-425f-408b-b575-11bbe25511b7 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:47.781114575 +0000 UTC m=+48.633497717 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/28db3bec-425f-408b-b575-11bbe25511b7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmsfd" (UID: "28db3bec-425f-408b-b575-11bbe25511b7") : secret "insights-runtime-extractor-tls" not found Jul 14 15:21:46.983371 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.983331 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-jhjxc" event={"ID":"cb4ec4df-6611-41c7-b1aa-f32a58b3a251","Type":"ContainerStarted","Data":"fe0bced6c6684183d45e7af3afeafecd8733cf4ce033932687309fd09840c731"} Jul 14 15:21:46.983512 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:46.983402 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-jhjxc" event={"ID":"cb4ec4df-6611-41c7-b1aa-f32a58b3a251","Type":"ContainerStarted","Data":"e08a4c1b3d94a9d973e65af444631ac36d72b8979d77c9c832d0d70a8a8228b1"} Jul 14 15:21:47.001934 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.001894 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-jhjxc" podStartSLOduration=2.190161367 podStartE2EDuration="4.001878457s" totalImagesPullingTime="1.81171709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:47.001622502 +0000 UTC m=+47.854005670" watchObservedRunningTime="2026-07-14 15:21:47.001878457 +0000 UTC m=+47.854261619" Jul 14 15:21:47.084057 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.084021 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:47.154097 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.154063 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:47.316515 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.316438 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-99pn4"] Jul 14 15:21:47.336462 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.336435 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-99pn4"] Jul 14 15:21:47.336586 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.336523 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-99pn4" Jul 14 15:21:47.339598 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.339575 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 14 15:21:47.339719 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.339607 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 14 15:21:47.339719 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.339687 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-2dhxd\"" Jul 14 15:21:47.341061 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.341044 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 14 15:21:47.341160 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.341044 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 14 15:21:47.353907 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.353884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:47.386272 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.386244 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1326b49f-329e-4064-baf2-11a8e840b2e0-signing-cabundle\") pod \"service-ca-79f68b9f4f-99pn4\" (UID: \"1326b49f-329e-4064-baf2-11a8e840b2e0\") " pod="openshift-service-ca/service-ca-79f68b9f4f-99pn4" Jul 14 15:21:47.386414 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.386393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5l4g\" (UniqueName: \"kubernetes.io/projected/1326b49f-329e-4064-baf2-11a8e840b2e0-kube-api-access-t5l4g\") pod \"service-ca-79f68b9f4f-99pn4\" (UID: \"1326b49f-329e-4064-baf2-11a8e840b2e0\") " pod="openshift-service-ca/service-ca-79f68b9f4f-99pn4" Jul 14 15:21:47.386467 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.386450 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1326b49f-329e-4064-baf2-11a8e840b2e0-signing-key\") pod \"service-ca-79f68b9f4f-99pn4\" (UID: \"1326b49f-329e-4064-baf2-11a8e840b2e0\") " pod="openshift-service-ca/service-ca-79f68b9f4f-99pn4" Jul 14 15:21:47.487776 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.487738 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1326b49f-329e-4064-baf2-11a8e840b2e0-signing-cabundle\") pod \"service-ca-79f68b9f4f-99pn4\" (UID: \"1326b49f-329e-4064-baf2-11a8e840b2e0\") " pod="openshift-service-ca/service-ca-79f68b9f4f-99pn4" Jul 14 15:21:47.487941 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.487860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5l4g\" (UniqueName: \"kubernetes.io/projected/1326b49f-329e-4064-baf2-11a8e840b2e0-kube-api-access-t5l4g\") pod \"service-ca-79f68b9f4f-99pn4\" (UID: \"1326b49f-329e-4064-baf2-11a8e840b2e0\") " pod="openshift-service-ca/service-ca-79f68b9f4f-99pn4" Jul 14 15:21:47.487941 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.487887 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1326b49f-329e-4064-baf2-11a8e840b2e0-signing-key\") pod \"service-ca-79f68b9f4f-99pn4\" (UID: \"1326b49f-329e-4064-baf2-11a8e840b2e0\") " pod="openshift-service-ca/service-ca-79f68b9f4f-99pn4" Jul 14 15:21:47.488571 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.488549 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1326b49f-329e-4064-baf2-11a8e840b2e0-signing-cabundle\") pod \"service-ca-79f68b9f4f-99pn4\" (UID: \"1326b49f-329e-4064-baf2-11a8e840b2e0\") " pod="openshift-service-ca/service-ca-79f68b9f4f-99pn4" Jul 14 15:21:47.490350 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.490326 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1326b49f-329e-4064-baf2-11a8e840b2e0-signing-key\") pod \"service-ca-79f68b9f4f-99pn4\" (UID: \"1326b49f-329e-4064-baf2-11a8e840b2e0\") " pod="openshift-service-ca/service-ca-79f68b9f4f-99pn4" Jul 14 15:21:47.497478 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.497458 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5l4g\" (UniqueName: \"kubernetes.io/projected/1326b49f-329e-4064-baf2-11a8e840b2e0-kube-api-access-t5l4g\") pod \"service-ca-79f68b9f4f-99pn4\" (UID: \"1326b49f-329e-4064-baf2-11a8e840b2e0\") " pod="openshift-service-ca/service-ca-79f68b9f4f-99pn4" Jul 14 15:21:47.645784 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.645747 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-99pn4" Jul 14 15:21:47.724116 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.724079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:47.782485 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.782450 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:47.790489 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.790462 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-99pn4"] Jul 14 15:21:47.790852 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.790832 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/28db3bec-425f-408b-b575-11bbe25511b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:47.790916 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.790902 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/28db3bec-425f-408b-b575-11bbe25511b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:47.791003 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:47.790985 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 15:21:47.791003 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:47.790998 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 15:21:47.791086 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:47.791048 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28db3bec-425f-408b-b575-11bbe25511b7-insights-runtime-extractor-tls podName:28db3bec-425f-408b-b575-11bbe25511b7 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:49.791034633 +0000 UTC m=+50.643417776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/28db3bec-425f-408b-b575-11bbe25511b7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmsfd" (UID: "28db3bec-425f-408b-b575-11bbe25511b7") : secret "insights-runtime-extractor-tls" not found Jul 14 15:21:47.791086 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:47.791061 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28db3bec-425f-408b-b575-11bbe25511b7-kube-rbac-proxy-cm podName:28db3bec-425f-408b-b575-11bbe25511b7 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:49.79105446 +0000 UTC m=+50.643437603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/28db3bec-425f-408b-b575-11bbe25511b7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmsfd" (UID: "28db3bec-425f-408b-b575-11bbe25511b7") : configmap "kube-rbac-proxy" not found Jul 14 15:21:47.793344 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:21:47.793325 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1326b49f_329e_4064_baf2_11a8e840b2e0.slice/crio-1be9c0bb70ed49a21336ea8b43a206b685b4fac1464531d772d27283a54c4a5b WatchSource:0}: Error finding container 1be9c0bb70ed49a21336ea8b43a206b685b4fac1464531d772d27283a54c4a5b: Status 404 returned error can't find the container with id 1be9c0bb70ed49a21336ea8b43a206b685b4fac1464531d772d27283a54c4a5b Jul 14 15:21:47.991145 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.991102 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-99pn4" event={"ID":"1326b49f-329e-4064-baf2-11a8e840b2e0","Type":"ContainerStarted","Data":"7870d56b35e91d0be7091e4318b04d71d169c39d710f90ebf863ac3c2551aebe"} Jul 14 15:21:47.991319 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:47.991156 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-99pn4" event={"ID":"1326b49f-329e-4064-baf2-11a8e840b2e0","Type":"ContainerStarted","Data":"1be9c0bb70ed49a21336ea8b43a206b685b4fac1464531d772d27283a54c4a5b"} Jul 14 15:21:48.010330 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:48.010277 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-99pn4" podStartSLOduration=1.010261662 podStartE2EDuration="1.010261662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:48.007949055 +0000 UTC m=+48.860332223" watchObservedRunningTime="2026-07-14 15:21:48.010261662 +0000 UTC m=+48.862644827" Jul 14 15:21:48.068424 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:48.068398 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jj7cz_db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb/dns-node-resolver/0.log" Jul 14 15:21:48.223444 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:48.223355 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:49.261772 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:49.261742 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-km9wq_3f3999dd-ca70-4115-8778-3c73e9f14124/node-ca/0.log" Jul 14 15:21:49.514776 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:49.514688 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:49.649194 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:49.649150 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:49.661192 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:49.661100 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xrzdx_f022e7cf-42aa-4780-b3f0-6004f70ae107/serve-healthcheck-canary/0.log" Jul 14 15:21:49.769537 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:49.769445 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:49.810627 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:49.810583 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/28db3bec-425f-408b-b575-11bbe25511b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:49.810766 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:49.810754 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/28db3bec-425f-408b-b575-11bbe25511b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:49.810882 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:49.810865 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 15:21:49.811083 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:49.810950 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 15:21:49.811198 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:49.811070 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28db3bec-425f-408b-b575-11bbe25511b7-kube-rbac-proxy-cm podName:28db3bec-425f-408b-b575-11bbe25511b7 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:53.811049723 +0000 UTC m=+54.663432870 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/28db3bec-425f-408b-b575-11bbe25511b7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmsfd" (UID: "28db3bec-425f-408b-b575-11bbe25511b7") : configmap "kube-rbac-proxy" not found Jul 14 15:21:49.811284 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:49.811236 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28db3bec-425f-408b-b575-11bbe25511b7-insights-runtime-extractor-tls podName:28db3bec-425f-408b-b575-11bbe25511b7 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:53.811218466 +0000 UTC m=+54.663601612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/28db3bec-425f-408b-b575-11bbe25511b7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmsfd" (UID: "28db3bec-425f-408b-b575-11bbe25511b7") : secret "insights-runtime-extractor-tls" not found Jul 14 15:21:49.862620 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:49.862594 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-jhjxc_cb4ec4df-6611-41c7-b1aa-f32a58b3a251/migrator/0.log" Jul 14 15:21:49.870224 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:49.870150 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:49.971120 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:49.971071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:50.062714 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:50.062642 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-jhjxc_cb4ec4df-6611-41c7-b1aa-f32a58b3a251/graceful-termination/0.log" Jul 14 15:21:50.263594 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:50.263539 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-jkhlz_2d1490f8-b484-46ee-83d6-4bb7fae67fd8/kube-storage-version-migrator-operator/0.log" Jul 14 15:21:50.517291 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:50.517258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:50.517291 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:50.517298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:21:50.517545 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:50.517322 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-4xn55\" (UID: \"02191573-3edc-4fe3-97af-d478beb7575c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-4xn55" Jul 14 15:21:50.517545 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:50.517357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-96wbh\" (UID: \"dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh" Jul 14 15:21:50.517545 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:50.517391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:21:50.517545 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:50.517437 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle podName:4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c nodeName:}" failed. No retries permitted until 2026-07-14 15:22:06.517417387 +0000 UTC m=+67.369800533 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle") pod "router-default-6cd7778556-bmkdk" (UID: "4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c") : configmap references non-existent config key: service-ca.crt Jul 14 15:21:50.517545 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:50.517443 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 15:21:50.517545 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:50.517470 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf4cb94c7-wmnzm: secret "image-registry-tls" not found Jul 14 15:21:50.517545 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:50.517482 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf70d46f-232a-48f2-bb18-7050cfc08c69-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wtdm4\" (UID: \"cf70d46f-232a-48f2-bb18-7050cfc08c69\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdm4" Jul 14 15:21:50.517545 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:50.517488 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 15:21:50.517545 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:50.517523 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls podName:e4db9b13-7375-4fbc-bd78-ed7b42e75f2c nodeName:}" failed. No retries permitted until 2026-07-14 15:22:06.517505301 +0000 UTC m=+67.369888447 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls") pod "image-registry-5bf4cb94c7-wmnzm" (UID: "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c") : secret "image-registry-tls" not found Jul 14 15:21:50.517545 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:50.517515 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 15:21:50.517545 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:50.517437 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 15:21:50.517545 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:50.517552 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs podName:4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c nodeName:}" failed. No retries permitted until 2026-07-14 15:22:06.517538448 +0000 UTC m=+67.369921591 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs") pod "router-default-6cd7778556-bmkdk" (UID: "4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c") : secret "router-metrics-certs-default" not found Jul 14 15:21:50.518221 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:50.517567 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls\") pod \"dns-default-lznzq\" (UID: \"d94ec422-3618-4b68-ae06-fc9232319c6d\") " pod="openshift-dns/dns-default-lznzq" Jul 14 15:21:50.518221 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:50.517593 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls podName:dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2 nodeName:}" failed. No retries permitted until 2026-07-14 15:22:06.517572961 +0000 UTC m=+67.369956108 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-96wbh" (UID: "dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2") : secret "cluster-monitoring-operator-tls" not found Jul 14 15:21:50.518221 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:50.517658 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 15:21:50.518221 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:50.517706 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls podName:d94ec422-3618-4b68-ae06-fc9232319c6d nodeName:}" failed. No retries permitted until 2026-07-14 15:22:06.5176937 +0000 UTC m=+67.370076844 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls") pod "dns-default-lznzq" (UID: "d94ec422-3618-4b68-ae06-fc9232319c6d") : secret "dns-default-metrics-tls" not found Jul 14 15:21:50.518221 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:50.518054 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert podName:02191573-3edc-4fe3-97af-d478beb7575c nodeName:}" failed. No retries permitted until 2026-07-14 15:22:06.518029898 +0000 UTC m=+67.370413045 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-4xn55" (UID: "02191573-3edc-4fe3-97af-d478beb7575c") : secret "networking-console-plugin-cert" not found Jul 14 15:21:50.533055 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:50.533033 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf70d46f-232a-48f2-bb18-7050cfc08c69-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wtdm4\" (UID: \"cf70d46f-232a-48f2-bb18-7050cfc08c69\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdm4" Jul 14 15:21:50.539092 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:50.539073 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdm4" Jul 14 15:21:50.562160 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:50.562132 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:50.664867 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:50.664829 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:50.692949 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:50.692917 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdm4"] Jul 14 15:21:51.003155 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:51.003116 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdm4" event={"ID":"cf70d46f-232a-48f2-bb18-7050cfc08c69","Type":"ContainerStarted","Data":"d549e7645cffc78f606c79d3f600391783ddd52a6844005a0d6df9a923a8cafe"} Jul 14 15:21:51.223235 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:51.223199 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:51.418936 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:51.418891 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:51.613874 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:51.613828 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:52.011749 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:52.011667 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:52.113155 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:52.113103 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:52.423411 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:52.423369 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:52.815083 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:52.815006 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:52.952660 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:52.952616 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-b9cq8" Jul 14 15:21:52.953053 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:52.953037 2579 scope.go:117] "RemoveContainer" containerID="11e1b47fefc2f3369a8c4bc6b9284cc5fc21160f25956849a867e3adca597484" Jul 14 15:21:52.953260 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:52.953241 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-b9cq8_openshift-console-operator(0c30c26a-ed99-46f7-89d8-020e9ed3a7c8)\"" pod="openshift-console-operator/console-operator-7866497494-b9cq8" podUID="0c30c26a-ed99-46f7-89d8-020e9ed3a7c8" Jul 14 15:21:53.231816 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:53.231777 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:53.457711 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:53.457621 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:53.702368 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:53.702329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:53.813766 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:53.813674 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:53.853845 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:53.853816 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/28db3bec-425f-408b-b575-11bbe25511b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:53.853984 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:53.853962 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/28db3bec-425f-408b-b575-11bbe25511b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:53.854050 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:53.854007 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 15:21:53.854104 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:21:53.854091 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28db3bec-425f-408b-b575-11bbe25511b7-insights-runtime-extractor-tls podName:28db3bec-425f-408b-b575-11bbe25511b7 nodeName:}" failed. No retries permitted until 2026-07-14 15:22:01.854069393 +0000 UTC m=+62.706452560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/28db3bec-425f-408b-b575-11bbe25511b7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmsfd" (UID: "28db3bec-425f-408b-b575-11bbe25511b7") : secret "insights-runtime-extractor-tls" not found Jul 14 15:21:53.854495 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:53.854474 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/28db3bec-425f-408b-b575-11bbe25511b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:21:54.012451 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:54.012395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:54.016184 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:54.016140 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdm4" event={"ID":"cf70d46f-232a-48f2-bb18-7050cfc08c69","Type":"ContainerStarted","Data":"ee672ea675f1ea59014c40af28fe81694eb3a2c87d34837f7cfbf069f1221105"} Jul 14 15:21:54.016314 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:54.016214 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdm4" event={"ID":"cf70d46f-232a-48f2-bb18-7050cfc08c69","Type":"ContainerStarted","Data":"d9ae7593f010b459c40ef335a913ba12465aa7ed4cf7fdb98ceb82a3333154c8"} Jul 14 15:21:54.036910 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:54.036864 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdm4" podStartSLOduration=37.859429431 podStartE2EDuration="40.036846172s" totalImagesPullingTime="2.177416741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:54.035960967 +0000 UTC m=+54.888344133" watchObservedRunningTime="2026-07-14 15:21:54.036846172 +0000 UTC m=+54.889229383" Jul 14 15:21:54.067213 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:54.067113 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:54.415765 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:54.415726 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:54.672846 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:54.672756 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:54.913789 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:54.913746 2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-b9cq8" Jul 14 15:21:54.914098 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:54.914085 2579 scope.go:117] "RemoveContainer" containerID="11e1b47fefc2f3369a8c4bc6b9284cc5fc21160f25956849a867e3adca597484" Jul 14 15:21:55.020029 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:55.019989 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:55.023642 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:55.023614 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 15:21:55.023778 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:55.023744 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-b9cq8" event={"ID":"0c30c26a-ed99-46f7-89d8-020e9ed3a7c8","Type":"ContainerStarted","Data":"c0f360932ff49e77db3be2bc3194d154ff50a3753ac738d64880f7b9fff766e9"} Jul 14 15:21:55.024310 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:55.024180 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-b9cq8" Jul 14 15:21:55.024896 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:55.024879 2579 patch_prober.go:28] interesting pod/console-operator-7866497494-b9cq8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.8:8443/readyz\": dial tcp 10.132.0.8:8443: connect: connection refused" start-of-body= Jul 14 15:21:55.024972 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:55.024919 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7866497494-b9cq8" podUID="0c30c26a-ed99-46f7-89d8-020e9ed3a7c8" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.8:8443/readyz\": dial tcp 10.132.0.8:8443: connect: connection refused" Jul 14 15:21:55.416863 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:55.416826 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:55.714305 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:55.714221 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:55.822278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:55.822240 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:56.030826 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:56.030718 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-b9cq8" Jul 14 15:21:56.065650 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:56.065612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:56.121627 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:56.121595 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:56.413125 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:56.413089 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:56.513591 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:56.513559 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:56.566795 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:56.566758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:56.687250 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:56.687159 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:56.858305 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:56.858266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:56.912696 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:56.912661 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:57.017049 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:57.016961 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:57.614036 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:57.613993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:59.271302 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:59.271256 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:59.337891 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:59.337861 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:59.491737 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:59.491703 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:21:59.812822 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:21:59.812788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:00.016301 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:00.016267 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:00.413600 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:00.413563 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:00.610333 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:00.610296 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:01.901261 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:01.901229 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9j4v" Jul 14 15:22:01.930200 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:01.930147 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/28db3bec-425f-408b-b575-11bbe25511b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:22:01.932560 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:01.932528 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/28db3bec-425f-408b-b575-11bbe25511b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsfd\" (UID: \"28db3bec-425f-408b-b575-11bbe25511b7\") " pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:22:01.994913 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:01.994888 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bllnj\"" Jul 14 15:22:02.002470 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:02.002454 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xmsfd" Jul 14 15:22:02.131402 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:02.131368 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xmsfd"] Jul 14 15:22:02.133795 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:22:02.133768 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28db3bec_425f_408b_b575_11bbe25511b7.slice/crio-6d6ad2bca12066d04129bfd9e3dff0fbcdfea2bad11d5a532963ab678ef43e43 WatchSource:0}: Error finding container 6d6ad2bca12066d04129bfd9e3dff0fbcdfea2bad11d5a532963ab678ef43e43: Status 404 returned error can't find the container with id 6d6ad2bca12066d04129bfd9e3dff0fbcdfea2bad11d5a532963ab678ef43e43 Jul 14 15:22:03.055914 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:03.055819 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmsfd" event={"ID":"28db3bec-425f-408b-b575-11bbe25511b7","Type":"ContainerStarted","Data":"6d6ad2bca12066d04129bfd9e3dff0fbcdfea2bad11d5a532963ab678ef43e43"} Jul 14 15:22:03.893840 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:03.893799 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:03.948247 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:03.948208 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:04.352221 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:04.352160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs\") pod \"network-metrics-daemon-kg2pm\" (UID: \"edc388f1-89b5-45bf-9672-3e1a135fed4e\") " pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:22:04.355395 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:04.355367 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 15:22:04.365602 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:04.365573 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc388f1-89b5-45bf-9672-3e1a135fed4e-metrics-certs\") pod \"network-metrics-daemon-kg2pm\" (UID: \"edc388f1-89b5-45bf-9672-3e1a135fed4e\") " pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:22:04.417022 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:04.416995 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vcjnm\"" Jul 14 15:22:04.424621 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:04.424592 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kg2pm" Jul 14 15:22:04.452737 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:04.452710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gw7t\" (UniqueName: \"kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t\") pod \"network-check-target-5lbpx\" (UID: \"edffdada-2745-451b-bf87-3ff5facb7b64\") " pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:22:04.452896 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:04.452813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret\") pod \"global-pull-secret-syncer-785zz\" (UID: \"f42d7bd8-f742-4b0a-83d9-2e0f121bda84\") " pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:22:04.455664 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:04.455635 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gw7t\" (UniqueName: \"kubernetes.io/projected/edffdada-2745-451b-bf87-3ff5facb7b64-kube-api-access-8gw7t\") pod \"network-check-target-5lbpx\" (UID: \"edffdada-2745-451b-bf87-3ff5facb7b64\") " pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:22:04.456115 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:04.456097 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 15:22:04.466109 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:04.466082 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f42d7bd8-f742-4b0a-83d9-2e0f121bda84-original-pull-secret\") pod \"global-pull-secret-syncer-785zz\" (UID: \"f42d7bd8-f742-4b0a-83d9-2e0f121bda84\") " pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:22:04.706955 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:04.706871 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zdwzw\"" Jul 14 15:22:04.708986 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:04.708963 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-785zz" Jul 14 15:22:04.714478 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:04.714449 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:22:05.295768 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:05.295729 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:05.398441 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:05.398401 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:05.921445 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:05.921402 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kg2pm"] Jul 14 15:22:05.927214 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:22:05.927162 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc388f1_89b5_45bf_9672_3e1a135fed4e.slice/crio-22578a9d43503b7d3cd899cf631705797b264aeaadce781dd05cc3fadafa4b2c WatchSource:0}: Error finding container 22578a9d43503b7d3cd899cf631705797b264aeaadce781dd05cc3fadafa4b2c: Status 404 returned error can't find the container with id 22578a9d43503b7d3cd899cf631705797b264aeaadce781dd05cc3fadafa4b2c Jul 14 15:22:05.935570 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:05.935536 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5lbpx"] Jul 14 15:22:05.939828 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:22:05.939800 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedffdada_2745_451b_bf87_3ff5facb7b64.slice/crio-e162e3781911637d57fb0495e46f9cf52c7dd4f4eb0aa9c241a098aab77822c5 WatchSource:0}: Error finding container e162e3781911637d57fb0495e46f9cf52c7dd4f4eb0aa9c241a098aab77822c5: Status 404 returned error can't find the container with id e162e3781911637d57fb0495e46f9cf52c7dd4f4eb0aa9c241a098aab77822c5 Jul 14 15:22:05.955250 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:05.955228 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-785zz"] Jul 14 15:22:06.067506 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.067431 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kg2pm" event={"ID":"edc388f1-89b5-45bf-9672-3e1a135fed4e","Type":"ContainerStarted","Data":"22578a9d43503b7d3cd899cf631705797b264aeaadce781dd05cc3fadafa4b2c"} Jul 14 15:22:06.068603 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.068567 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5lbpx" event={"ID":"edffdada-2745-451b-bf87-3ff5facb7b64","Type":"ContainerStarted","Data":"e162e3781911637d57fb0495e46f9cf52c7dd4f4eb0aa9c241a098aab77822c5"} Jul 14 15:22:06.129213 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:22:06.129158 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42d7bd8_f742_4b0a_83d9_2e0f121bda84.slice/crio-361a352d790b780752fc630c85c3b89ab0df6dad7252e6b85240fc4b6dab2cec WatchSource:0}: Error finding container 361a352d790b780752fc630c85c3b89ab0df6dad7252e6b85240fc4b6dab2cec: Status 404 returned error can't find the container with id 361a352d790b780752fc630c85c3b89ab0df6dad7252e6b85240fc4b6dab2cec Jul 14 15:22:06.571503 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.571464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:22:06.571935 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.571526 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:22:06.571935 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.571565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-4xn55\" (UID: \"02191573-3edc-4fe3-97af-d478beb7575c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-4xn55" Jul 14 15:22:06.571935 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.571602 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-96wbh\" (UID: \"dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh" Jul 14 15:22:06.571935 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.571656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:22:06.571935 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.571700 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls\") pod \"dns-default-lznzq\" (UID: \"d94ec422-3618-4b68-ae06-fc9232319c6d\") " pod="openshift-dns/dns-default-lznzq" Jul 14 15:22:06.573059 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.573009 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-service-ca-bundle\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:22:06.574957 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.574912 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d94ec422-3618-4b68-ae06-fc9232319c6d-metrics-tls\") pod \"dns-default-lznzq\" (UID: \"d94ec422-3618-4b68-ae06-fc9232319c6d\") " pod="openshift-dns/dns-default-lznzq" Jul 14 15:22:06.575075 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.575025 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/02191573-3edc-4fe3-97af-d478beb7575c-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-4xn55\" (UID: \"02191573-3edc-4fe3-97af-d478beb7575c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-4xn55" Jul 14 15:22:06.575592 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.575569 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-96wbh\" (UID: \"dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh" Jul 14 15:22:06.575822 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.575800 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls\") pod \"image-registry-5bf4cb94c7-wmnzm\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:22:06.577696 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.577674 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c-metrics-certs\") pod \"router-default-6cd7778556-bmkdk\" (UID: \"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c\") " pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:22:06.614294 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.614264 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bf4cb94c7-wmnzm"] Jul 14 15:22:06.617269 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.617244 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-twn9v\"" Jul 14 15:22:06.624860 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.624836 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:22:06.670776 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.670746 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-574655b545-ggbxf"] Jul 14 15:22:06.698975 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.698940 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:06.703336 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.702872 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-574655b545-ggbxf"] Jul 14 15:22:06.703336 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.703031 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.750889 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.750855 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-wnzh4\"" Jul 14 15:22:06.758478 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.758455 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-4xn55" Jul 14 15:22:06.759900 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.759881 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-n6jhf\"" Jul 14 15:22:06.767672 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.767648 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:22:06.781574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.781550 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-48k9t\"" Jul 14 15:22:06.789807 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.789787 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh" Jul 14 15:22:06.800815 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.800792 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xtkj9\"" Jul 14 15:22:06.808419 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.808394 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lznzq" Jul 14 15:22:06.816951 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.816919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:06.817428 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.817410 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bf4cb94c7-wmnzm"] Jul 14 15:22:06.873684 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.873647 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28a71623-814f-4087-8206-4b0baca8e043-trusted-ca\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.873861 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.873697 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28a71623-814f-4087-8206-4b0baca8e043-bound-sa-token\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.873861 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.873726 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28a71623-814f-4087-8206-4b0baca8e043-registry-tls\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.873861 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.873767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28a71623-814f-4087-8206-4b0baca8e043-ca-trust-extracted\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.873861 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.873846 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28a71623-814f-4087-8206-4b0baca8e043-image-registry-private-configuration\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.874069 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.873889 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28a71623-814f-4087-8206-4b0baca8e043-registry-certificates\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.874069 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.873918 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k7gh\" (UniqueName: \"kubernetes.io/projected/28a71623-814f-4087-8206-4b0baca8e043-kube-api-access-9k7gh\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.874069 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.873968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28a71623-814f-4087-8206-4b0baca8e043-installation-pull-secrets\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.909743 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.909692 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:06.975087 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.975052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28a71623-814f-4087-8206-4b0baca8e043-installation-pull-secrets\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.975087 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.975094 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28a71623-814f-4087-8206-4b0baca8e043-trusted-ca\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.975369 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.975113 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28a71623-814f-4087-8206-4b0baca8e043-bound-sa-token\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.975369 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.975215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28a71623-814f-4087-8206-4b0baca8e043-registry-tls\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.975369 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.975275 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28a71623-814f-4087-8206-4b0baca8e043-ca-trust-extracted\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.975369 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.975322 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28a71623-814f-4087-8206-4b0baca8e043-image-registry-private-configuration\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.975369 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.975358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28a71623-814f-4087-8206-4b0baca8e043-registry-certificates\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.975617 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.975378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k7gh\" (UniqueName: \"kubernetes.io/projected/28a71623-814f-4087-8206-4b0baca8e043-kube-api-access-9k7gh\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.975671 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.975645 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28a71623-814f-4087-8206-4b0baca8e043-ca-trust-extracted\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.976241 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.976215 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28a71623-814f-4087-8206-4b0baca8e043-trusted-ca\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.976386 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.976286 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28a71623-814f-4087-8206-4b0baca8e043-registry-certificates\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.978105 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.978085 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28a71623-814f-4087-8206-4b0baca8e043-installation-pull-secrets\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.978740 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.978715 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28a71623-814f-4087-8206-4b0baca8e043-registry-tls\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.978839 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.978723 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28a71623-814f-4087-8206-4b0baca8e043-image-registry-private-configuration\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.993988 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.993964 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28a71623-814f-4087-8206-4b0baca8e043-bound-sa-token\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:06.994113 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:06.994088 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k7gh\" (UniqueName: \"kubernetes.io/projected/28a71623-814f-4087-8206-4b0baca8e043-kube-api-access-9k7gh\") pod \"image-registry-574655b545-ggbxf\" (UID: \"28a71623-814f-4087-8206-4b0baca8e043\") " pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:07.007917 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:07.007878 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:07.014868 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:07.014847 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:07.061263 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:22:07.061218 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4db9b13_7375_4fbc_bd78_ed7b42e75f2c.slice/crio-c14e5279b3fe7c07bff24a062d3f286cf9d176e201f0ee6bbb8f48ace6355b00 WatchSource:0}: Error finding container c14e5279b3fe7c07bff24a062d3f286cf9d176e201f0ee6bbb8f48ace6355b00: Status 404 returned error can't find the container with id c14e5279b3fe7c07bff24a062d3f286cf9d176e201f0ee6bbb8f48ace6355b00 Jul 14 15:22:07.081001 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:07.080920 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5lbpx" event={"ID":"edffdada-2745-451b-bf87-3ff5facb7b64","Type":"ContainerStarted","Data":"dd920d7277230897b736ffb63c03e178927d0767b0e52b1848dfd1e1a9f9cd55"} Jul 14 15:22:07.082423 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:07.082389 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" event={"ID":"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c","Type":"ContainerStarted","Data":"c14e5279b3fe7c07bff24a062d3f286cf9d176e201f0ee6bbb8f48ace6355b00"} Jul 14 15:22:07.084269 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:07.084214 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-785zz" event={"ID":"f42d7bd8-f742-4b0a-83d9-2e0f121bda84","Type":"ContainerStarted","Data":"361a352d790b780752fc630c85c3b89ab0df6dad7252e6b85240fc4b6dab2cec"} Jul 14 15:22:07.086965 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:07.086755 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmsfd" event={"ID":"28db3bec-425f-408b-b575-11bbe25511b7","Type":"ContainerStarted","Data":"551249c909afc270e99e2be6cb0ba0e3502329c81682b54d46871b553237f4f8"} Jul 14 15:22:07.102579 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:07.102515 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:22:07.122285 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:07.122220 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5lbpx" podStartSLOduration=68.122195917 podStartE2EDuration="1m8.122195917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:07.120873821 +0000 UTC m=+67.973256987" watchObservedRunningTime="2026-07-14 15:22:07.122195917 +0000 UTC m=+67.974579062" Jul 14 15:22:07.178550 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:07.174900 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:07.241071 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:07.240821 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:07.302499 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:07.302461 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lznzq"] Jul 14 15:22:07.322511 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:22:07.322464 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94ec422_3618_4b68_ae06_fc9232319c6d.slice/crio-7ec5c8f5d4642c73a6905f225bece419cd4acd1435833fe8ec9ff954e1175f60 WatchSource:0}: Error finding container 7ec5c8f5d4642c73a6905f225bece419cd4acd1435833fe8ec9ff954e1175f60: Status 404 returned error can't find the container with id 7ec5c8f5d4642c73a6905f225bece419cd4acd1435833fe8ec9ff954e1175f60 Jul 14 15:22:07.337690 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:07.337606 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh"] Jul 14 15:22:07.356225 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:07.356138 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-574655b545-ggbxf"] Jul 14 15:22:07.375456 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:07.375427 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-4xn55"] Jul 14 15:22:07.378161 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:07.378100 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6cd7778556-bmkdk"] Jul 14 15:22:07.429104 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:22:07.428966 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a71623_814f_4087_8206_4b0baca8e043.slice/crio-5fb1dd0b1a262df1dcffa65993bf2857341a6bc8d53a2b091a95656d62c2ba29 WatchSource:0}: Error finding container 5fb1dd0b1a262df1dcffa65993bf2857341a6bc8d53a2b091a95656d62c2ba29: Status 404 returned error can't find the container with id 5fb1dd0b1a262df1dcffa65993bf2857341a6bc8d53a2b091a95656d62c2ba29 Jul 14 15:22:07.429408 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:22:07.429374 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4a3f8d_bceb_4bff_ac61_3b3bc77e28c2.slice/crio-47ce564a0cb3b1d6417208cf78b4e4140ada5bdb20cba250f9115139cf051e6b WatchSource:0}: Error finding container 47ce564a0cb3b1d6417208cf78b4e4140ada5bdb20cba250f9115139cf051e6b: Status 404 returned error can't find the container with id 47ce564a0cb3b1d6417208cf78b4e4140ada5bdb20cba250f9115139cf051e6b Jul 14 15:22:07.430527 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:22:07.430501 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a90d0a9_4d08_4c0d_abfe_dadd776ecc8c.slice/crio-59604165fc05a29e3519848c58c5d2b3c69eec07e0bc9641f61ca7d23c551165 WatchSource:0}: Error finding container 59604165fc05a29e3519848c58c5d2b3c69eec07e0bc9641f61ca7d23c551165: Status 404 returned error can't find the container with id 59604165fc05a29e3519848c58c5d2b3c69eec07e0bc9641f61ca7d23c551165 Jul 14 15:22:08.093156 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.093112 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh" event={"ID":"dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2","Type":"ContainerStarted","Data":"47ce564a0cb3b1d6417208cf78b4e4140ada5bdb20cba250f9115139cf051e6b"} Jul 14 15:22:08.094612 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.094579 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-4xn55" event={"ID":"02191573-3edc-4fe3-97af-d478beb7575c","Type":"ContainerStarted","Data":"cf18c9bc865b36c867a90c514d099d5ac89f013cbcad603eda614d9caa344a06"} Jul 14 15:22:08.096550 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.096519 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-574655b545-ggbxf" event={"ID":"28a71623-814f-4087-8206-4b0baca8e043","Type":"ContainerStarted","Data":"8e309a644e0ba38851913c4b25a04faaec2306dacb0afacaa205ccd3ab9e40b3"} Jul 14 15:22:08.096652 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.096570 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-574655b545-ggbxf" event={"ID":"28a71623-814f-4087-8206-4b0baca8e043","Type":"ContainerStarted","Data":"5fb1dd0b1a262df1dcffa65993bf2857341a6bc8d53a2b091a95656d62c2ba29"} Jul 14 15:22:08.096652 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.096612 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:08.098692 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.098668 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kg2pm" event={"ID":"edc388f1-89b5-45bf-9672-3e1a135fed4e","Type":"ContainerStarted","Data":"1f0ec7a3dfa3ac58fbe10326731549033f313ff7f98af98d6fa66d76e68303dc"} Jul 14 15:22:08.098798 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.098716 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kg2pm" event={"ID":"edc388f1-89b5-45bf-9672-3e1a135fed4e","Type":"ContainerStarted","Data":"8955ac0b8c693f3eb4265f83ab3dc94a2591122f7d49ab2d3ad5ab7d42c01efb"} Jul 14 15:22:08.100779 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.100683 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" event={"ID":"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c","Type":"ContainerStarted","Data":"7d5fa712427848f220b714b23abdbeb4d0c6de9d5b83be2822df8d56420909c2"} Jul 14 15:22:08.100779 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.100743 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:22:08.103201 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.103156 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmsfd" event={"ID":"28db3bec-425f-408b-b575-11bbe25511b7","Type":"ContainerStarted","Data":"f77cd040b502425700ea8f411de266888f239c933e47fc6736265d1f11a566e7"} Jul 14 15:22:08.104786 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.104737 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lznzq" event={"ID":"d94ec422-3618-4b68-ae06-fc9232319c6d","Type":"ContainerStarted","Data":"7ec5c8f5d4642c73a6905f225bece419cd4acd1435833fe8ec9ff954e1175f60"} Jul 14 15:22:08.106745 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.106721 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6cd7778556-bmkdk" event={"ID":"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c","Type":"ContainerStarted","Data":"25ec92f81d384c851f830b0a4e17512a80fb64c12353621cffecd5709e1a611e"} Jul 14 15:22:08.106848 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.106765 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6cd7778556-bmkdk" event={"ID":"4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c","Type":"ContainerStarted","Data":"59604165fc05a29e3519848c58c5d2b3c69eec07e0bc9641f61ca7d23c551165"} Jul 14 15:22:08.117453 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.117406 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-574655b545-ggbxf" podStartSLOduration=2.117389362 podStartE2EDuration="2.117389362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:08.116015264 +0000 UTC m=+68.968398430" watchObservedRunningTime="2026-07-14 15:22:08.117389362 +0000 UTC m=+68.969772530" Jul 14 15:22:08.141222 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.141145 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6cd7778556-bmkdk" podStartSLOduration=54.141126332 podStartE2EDuration="54.141126332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:08.140564468 +0000 UTC m=+68.992947634" watchObservedRunningTime="2026-07-14 15:22:08.141126332 +0000 UTC m=+68.993509478" Jul 14 15:22:08.157997 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.157946 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kg2pm" podStartSLOduration=67.648658966 podStartE2EDuration="1m9.157930873s" totalImagesPullingTime="1.509271907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:08.157004433 +0000 UTC m=+69.009387599" watchObservedRunningTime="2026-07-14 15:22:08.157930873 +0000 UTC m=+69.010314040" Jul 14 15:22:08.177364 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.177317 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" podStartSLOduration=54.177295754 podStartE2EDuration="54.177295754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:08.176585901 +0000 UTC m=+69.028969068" watchObservedRunningTime="2026-07-14 15:22:08.177295754 +0000 UTC m=+69.029678921" Jul 14 15:22:08.314279 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.314234 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:08.415240 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.415133 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:08.768430 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.768348 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:22:08.771269 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:08.771242 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:22:09.113323 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:09.113153 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-4xn55" event={"ID":"02191573-3edc-4fe3-97af-d478beb7575c","Type":"ContainerStarted","Data":"24e820f399e2ba7e6f55511cf162cf70834cbacf9b596202ef3ddc9d48da720e"} Jul 14 15:22:09.113732 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:09.113519 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:22:09.114879 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:09.114856 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6cd7778556-bmkdk" Jul 14 15:22:09.133345 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:09.133284 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-4xn55" podStartSLOduration=53.636592043 podStartE2EDuration="55.133266713s" totalImagesPullingTime="1.49667467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:09.130778625 +0000 UTC m=+69.983161810" watchObservedRunningTime="2026-07-14 15:22:09.133266713 +0000 UTC m=+69.985649884" Jul 14 15:22:09.280199 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:09.280148 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:09.384681 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:09.384545 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:09.693557 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:09.693474 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:09.755133 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:09.754923 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:10.136130 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:10.136081 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:11.292209 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:11.292152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:11.346526 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:11.346479 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:12.155708 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:12.155661 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:12.258790 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:12.258704 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:13.287698 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:13.287652 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:13.343232 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:13.343193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:13.942491 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:13.942453 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:14.043238 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:14.043204 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:14.137842 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:14.137805 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmsfd" event={"ID":"28db3bec-425f-408b-b575-11bbe25511b7","Type":"ContainerStarted","Data":"7734cb01fd33d08398af073cb3a742b15aa2fb08becbcde3339d115bf1d0d237"} Jul 14 15:22:14.139896 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:14.139868 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lznzq" event={"ID":"d94ec422-3618-4b68-ae06-fc9232319c6d","Type":"ContainerStarted","Data":"42726413076d0b99c4d5108c272aeb045ccda9fb228211b6ccde34f36a249d3c"} Jul 14 15:22:14.140030 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:14.139917 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lznzq" event={"ID":"d94ec422-3618-4b68-ae06-fc9232319c6d","Type":"ContainerStarted","Data":"ace915737db0989b3c22ce7d95c634dd2ac0fa490ae42cebc12837ff998a3e7c"} Jul 14 15:22:14.140030 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:14.139945 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lznzq" Jul 14 15:22:14.141547 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:14.141526 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh" event={"ID":"dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2","Type":"ContainerStarted","Data":"bb2be5fc8b9ff73d9488a0ab3d90bc77be79283a9ed76eaee471d2132f709ff4"} Jul 14 15:22:14.143155 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:14.143126 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-785zz" event={"ID":"f42d7bd8-f742-4b0a-83d9-2e0f121bda84","Type":"ContainerStarted","Data":"4b98dcd4352178a97d5a0508cc5fc7b35a4e8dac04e56d56c5c88a4bc7b01e0a"} Jul 14 15:22:14.164460 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:14.164433 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:14.172953 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:14.172911 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xmsfd" podStartSLOduration=16.692242484 podStartE2EDuration="28.172896737s" totalImagesPullingTime="11.480654253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:21:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:14.170599224 +0000 UTC m=+75.022982393" watchObservedRunningTime="2026-07-14 15:22:14.172896737 +0000 UTC m=+75.025279901" Jul 14 15:22:14.190223 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:14.190188 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-96wbh" podStartSLOduration=54.106989829 podStartE2EDuration="1m0.190153644s" totalImagesPullingTime="6.083163815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:14.189821259 +0000 UTC m=+75.042204426" watchObservedRunningTime="2026-07-14 15:22:14.190153644 +0000 UTC m=+75.042536809" Jul 14 15:22:14.212978 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:14.212932 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lznzq" podStartSLOduration=35.166068891 podStartE2EDuration="40.212918194s" totalImagesPullingTime="5.046849303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:14.21146064 +0000 UTC m=+75.063843807" watchObservedRunningTime="2026-07-14 15:22:14.212918194 +0000 UTC m=+75.065301359" Jul 14 15:22:14.235429 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:14.235385 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-785zz" podStartSLOduration=68.021376504 podStartE2EDuration="1m14.235369788s" totalImagesPullingTime="6.213993284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:21:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:14.234740371 +0000 UTC m=+75.087123535" watchObservedRunningTime="2026-07-14 15:22:14.235369788 +0000 UTC m=+75.087752954" Jul 14 15:22:14.646988 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:14.646951 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:15.493224 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:15.493130 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:15.593977 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:15.593930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:16.358495 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:16.358459 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:16.417524 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:16.417492 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:17.375416 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:17.375383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:17.457002 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:17.456968 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:17.545996 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:17.545964 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:17.613059 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:17.613016 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:18.235503 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:18.235456 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:18.334485 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:18.334446 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:18.434281 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:18.434243 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:18.895738 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:18.895708 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:18.954311 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:18.954274 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:20.218752 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:20.218719 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:20.306984 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:20.306949 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:20.462039 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:20.461999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:21.274928 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:21.274894 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:21.376533 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:21.376502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:22.240088 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:22.240059 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:23.955257 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:23.955220 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:24.008826 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.008789 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:24.149473 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.149442 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lznzq" Jul 14 15:22:24.752677 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.752641 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8hrpm"] Jul 14 15:22:24.756867 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.756842 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.760188 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.760005 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 14 15:22:24.760188 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.760057 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 14 15:22:24.760188 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.760103 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 14 15:22:24.760937 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.760711 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k74s7\"" Jul 14 15:22:24.760937 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.760785 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 14 15:22:24.841014 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.840979 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1460f341-5423-4590-aad7-7bde6bbefe61-sys\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.841216 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.841032 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1460f341-5423-4590-aad7-7bde6bbefe61-node-exporter-accelerators-collector-config\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.841216 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.841066 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1460f341-5423-4590-aad7-7bde6bbefe61-node-exporter-tls\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.841216 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.841101 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1460f341-5423-4590-aad7-7bde6bbefe61-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.841216 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.841136 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1460f341-5423-4590-aad7-7bde6bbefe61-metrics-client-ca\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.841216 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.841167 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1460f341-5423-4590-aad7-7bde6bbefe61-root\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.841216 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.841207 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj7hl\" (UniqueName: \"kubernetes.io/projected/1460f341-5423-4590-aad7-7bde6bbefe61-kube-api-access-jj7hl\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.841507 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.841236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1460f341-5423-4590-aad7-7bde6bbefe61-node-exporter-wtmp\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.841507 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.841299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1460f341-5423-4590-aad7-7bde6bbefe61-node-exporter-textfile\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.942271 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.942230 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1460f341-5423-4590-aad7-7bde6bbefe61-metrics-client-ca\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.942271 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.942268 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1460f341-5423-4590-aad7-7bde6bbefe61-root\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.942495 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.942293 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj7hl\" (UniqueName: \"kubernetes.io/projected/1460f341-5423-4590-aad7-7bde6bbefe61-kube-api-access-jj7hl\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.942495 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.942313 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1460f341-5423-4590-aad7-7bde6bbefe61-node-exporter-wtmp\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.942495 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.942360 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1460f341-5423-4590-aad7-7bde6bbefe61-node-exporter-textfile\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.942495 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.942389 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1460f341-5423-4590-aad7-7bde6bbefe61-sys\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.942495 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.942407 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1460f341-5423-4590-aad7-7bde6bbefe61-node-exporter-accelerators-collector-config\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.942495 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.942432 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1460f341-5423-4590-aad7-7bde6bbefe61-node-exporter-tls\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.942495 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.942463 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1460f341-5423-4590-aad7-7bde6bbefe61-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.943922 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.942923 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1460f341-5423-4590-aad7-7bde6bbefe61-root\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.943922 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.942928 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1460f341-5423-4590-aad7-7bde6bbefe61-node-exporter-wtmp\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.943922 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.943527 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1460f341-5423-4590-aad7-7bde6bbefe61-metrics-client-ca\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.943922 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:22:24.943633 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 14 15:22:24.943922 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:22:24.943688 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1460f341-5423-4590-aad7-7bde6bbefe61-node-exporter-tls podName:1460f341-5423-4590-aad7-7bde6bbefe61 nodeName:}" failed. No retries permitted until 2026-07-14 15:22:25.44367029 +0000 UTC m=+86.296053437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1460f341-5423-4590-aad7-7bde6bbefe61-node-exporter-tls") pod "node-exporter-8hrpm" (UID: "1460f341-5423-4590-aad7-7bde6bbefe61") : secret "node-exporter-tls" not found Jul 14 15:22:24.943922 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.943775 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1460f341-5423-4590-aad7-7bde6bbefe61-node-exporter-accelerators-collector-config\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.944346 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.944009 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1460f341-5423-4590-aad7-7bde6bbefe61-node-exporter-textfile\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.944477 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.944451 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1460f341-5423-4590-aad7-7bde6bbefe61-sys\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.945722 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.945698 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1460f341-5423-4590-aad7-7bde6bbefe61-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:24.953653 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:24.953608 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj7hl\" (UniqueName: \"kubernetes.io/projected/1460f341-5423-4590-aad7-7bde6bbefe61-kube-api-access-jj7hl\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:25.446992 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:25.446958 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1460f341-5423-4590-aad7-7bde6bbefe61-node-exporter-tls\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:25.449518 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:25.449491 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1460f341-5423-4590-aad7-7bde6bbefe61-node-exporter-tls\") pod \"node-exporter-8hrpm\" (UID: \"1460f341-5423-4590-aad7-7bde6bbefe61\") " pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:25.623166 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:25.623124 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:25.669158 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:25.669130 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8hrpm" Jul 14 15:22:25.684546 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:22:25.684519 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1460f341_5423_4590_aad7_7bde6bbefe61.slice/crio-cbb760ce16bd7462debbe678ec49c2289b4418cb0bc0b8d6b27fb6cb3b343921 WatchSource:0}: Error finding container cbb760ce16bd7462debbe678ec49c2289b4418cb0bc0b8d6b27fb6cb3b343921: Status 404 returned error can't find the container with id cbb760ce16bd7462debbe678ec49c2289b4418cb0bc0b8d6b27fb6cb3b343921 Jul 14 15:22:25.720453 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:25.720368 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:25.794801 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:25.794761 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:26.199096 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:26.199055 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hrpm" event={"ID":"1460f341-5423-4590-aad7-7bde6bbefe61","Type":"ContainerStarted","Data":"cbb760ce16bd7462debbe678ec49c2289b4418cb0bc0b8d6b27fb6cb3b343921"} Jul 14 15:22:27.204524 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:27.204494 2579 generic.go:356] "Generic (PLEG): container finished" podID="1460f341-5423-4590-aad7-7bde6bbefe61" containerID="c37a41d82b21d858f2bb501e9bbc8ecfec1a57a28dca4191854ecaf724a101f4" exitCode=0 Jul 14 15:22:27.204886 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:27.204581 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hrpm" event={"ID":"1460f341-5423-4590-aad7-7bde6bbefe61","Type":"ContainerDied","Data":"c37a41d82b21d858f2bb501e9bbc8ecfec1a57a28dca4191854ecaf724a101f4"} Jul 14 15:22:27.657141 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:27.657090 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:27.749396 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:27.749362 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:27.898854 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:27.898813 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:28.106883 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:28.106856 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:22:28.210675 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:28.210632 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hrpm" event={"ID":"1460f341-5423-4590-aad7-7bde6bbefe61","Type":"ContainerStarted","Data":"83f0225135e997a914e066f65ee7ed426d0885aafaf4cabfd5cc2d0bae6592fd"} Jul 14 15:22:28.211150 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:28.210701 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hrpm" event={"ID":"1460f341-5423-4590-aad7-7bde6bbefe61","Type":"ContainerStarted","Data":"617d04edc9230e050de30437e4c2981ed02d5376f21e3ff86aa261a3cac513da"} Jul 14 15:22:28.231333 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:28.231290 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8hrpm" podStartSLOduration=3.223676716 podStartE2EDuration="4.231269672s" totalImagesPullingTime="934.935913ms" totalInitContainerRuntime="72.657043ms" isStatefulPod=true podCreationTimestamp="2026-07-14 15:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:28.229953899 +0000 UTC m=+89.082337064" watchObservedRunningTime="2026-07-14 15:22:28.231269672 +0000 UTC m=+89.083652835" Jul 14 15:22:28.261659 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:28.261629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:28.365480 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:28.365399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:28.783735 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:28.783652 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:28.884895 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:28.884863 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:28.993256 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:28.993223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:29.118138 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:29.118113 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-574655b545-ggbxf" Jul 14 15:22:29.265007 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:29.264975 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:29.366520 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:29.366481 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:29.464416 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:29.464323 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:29.577815 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:29.577773 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:29.647076 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:29.647028 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:29.976155 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:29.976120 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:30.036051 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:30.036017 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:30.972308 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:30.972268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:31.212241 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:31.212210 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:32.644520 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:32.644483 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:32.709748 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:32.709704 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:33.128224 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.128158 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" podUID="e4db9b13-7375-4fbc-bd78-ed7b42e75f2c" containerName="registry" containerID="cri-o://7d5fa712427848f220b714b23abdbeb4d0c6de9d5b83be2822df8d56420909c2" gracePeriod=30 Jul 14 15:22:33.369883 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.369856 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:22:33.520123 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.520035 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-installation-pull-secrets\") pod \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " Jul 14 15:22:33.520123 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.520100 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls\") pod \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " Jul 14 15:22:33.520123 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.520126 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-bound-sa-token\") pod \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " Jul 14 15:22:33.520416 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.520149 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6n5w\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-kube-api-access-p6n5w\") pod \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " Jul 14 15:22:33.520416 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.520165 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-trusted-ca\") pod \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " Jul 14 15:22:33.520416 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.520224 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-image-registry-private-configuration\") pod \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " Jul 14 15:22:33.520416 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.520270 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-ca-trust-extracted\") pod \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " Jul 14 15:22:33.520416 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.520299 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-certificates\") pod \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\" (UID: \"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c\") " Jul 14 15:22:33.520795 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.520762 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c" (UID: "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 15:22:33.520924 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.520838 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c" (UID: "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 15:22:33.522865 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.522819 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c" (UID: "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 15:22:33.523002 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.522975 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c" (UID: "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 15:22:33.523110 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.522950 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c" (UID: "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:22:33.523110 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.523041 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-kube-api-access-p6n5w" (OuterVolumeSpecName: "kube-api-access-p6n5w") pod "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c" (UID: "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c"). InnerVolumeSpecName "kube-api-access-p6n5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:22:33.523110 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.523086 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c" (UID: "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:22:33.529116 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.529092 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c" (UID: "e4db9b13-7375-4fbc-bd78-ed7b42e75f2c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:22:33.621597 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.621551 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-installation-pull-secrets\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:22:33.621597 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.621592 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-tls\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:22:33.621597 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.621601 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-bound-sa-token\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:22:33.621597 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.621608 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6n5w\" (UniqueName: \"kubernetes.io/projected/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-kube-api-access-p6n5w\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:22:33.621597 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.621618 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-trusted-ca\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:22:33.621873 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.621626 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-image-registry-private-configuration\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:22:33.621873 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.621635 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-ca-trust-extracted\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:22:33.621873 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:33.621644 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c-registry-certificates\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:22:34.239869 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:34.239830 2579 generic.go:356] "Generic (PLEG): container finished" podID="e4db9b13-7375-4fbc-bd78-ed7b42e75f2c" containerID="7d5fa712427848f220b714b23abdbeb4d0c6de9d5b83be2822df8d56420909c2" exitCode=0 Jul 14 15:22:34.240307 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:34.239884 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" Jul 14 15:22:34.240307 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:34.239886 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" event={"ID":"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c","Type":"ContainerDied","Data":"7d5fa712427848f220b714b23abdbeb4d0c6de9d5b83be2822df8d56420909c2"} Jul 14 15:22:34.240307 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:34.240003 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bf4cb94c7-wmnzm" event={"ID":"e4db9b13-7375-4fbc-bd78-ed7b42e75f2c","Type":"ContainerDied","Data":"c14e5279b3fe7c07bff24a062d3f286cf9d176e201f0ee6bbb8f48ace6355b00"} Jul 14 15:22:34.240307 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:34.240026 2579 scope.go:117] "RemoveContainer" containerID="7d5fa712427848f220b714b23abdbeb4d0c6de9d5b83be2822df8d56420909c2" Jul 14 15:22:34.248232 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:34.248212 2579 scope.go:117] "RemoveContainer" containerID="7d5fa712427848f220b714b23abdbeb4d0c6de9d5b83be2822df8d56420909c2" Jul 14 15:22:34.248494 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:22:34.248472 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d5fa712427848f220b714b23abdbeb4d0c6de9d5b83be2822df8d56420909c2\": container with ID starting with 7d5fa712427848f220b714b23abdbeb4d0c6de9d5b83be2822df8d56420909c2 not found: ID does not exist" containerID="7d5fa712427848f220b714b23abdbeb4d0c6de9d5b83be2822df8d56420909c2" Jul 14 15:22:34.248555 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:34.248503 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d5fa712427848f220b714b23abdbeb4d0c6de9d5b83be2822df8d56420909c2"} err="failed to get container status \"7d5fa712427848f220b714b23abdbeb4d0c6de9d5b83be2822df8d56420909c2\": rpc error: code = NotFound desc = could not find container \"7d5fa712427848f220b714b23abdbeb4d0c6de9d5b83be2822df8d56420909c2\": container with ID starting with 7d5fa712427848f220b714b23abdbeb4d0c6de9d5b83be2822df8d56420909c2 not found: ID does not exist" Jul 14 15:22:34.262858 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:34.262820 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bf4cb94c7-wmnzm"] Jul 14 15:22:34.264868 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:34.264846 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5bf4cb94c7-wmnzm"] Jul 14 15:22:35.696904 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:35.696856 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4db9b13-7375-4fbc-bd78-ed7b42e75f2c" path="/var/lib/kubelet/pods/e4db9b13-7375-4fbc-bd78-ed7b42e75f2c/volumes" Jul 14 15:22:38.109384 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:38.109352 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5lbpx" Jul 14 15:22:39.134044 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:39.134007 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:39.191342 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:39.191303 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:41.642773 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:41.642745 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:41.891549 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:41.891507 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:41.951733 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:41.951645 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:42.122567 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:42.122524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:43.272952 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:43.272919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:43.328063 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:43.328024 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:56.320684 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:56.320647 2579 generic.go:356] "Generic (PLEG): container finished" podID="cbb50f85-a13f-451d-a4bb-df7a9f212438" containerID="c7db711a12f1cec341845aa6429f0f92d8b39e5a26bf3dd046d959bb4acad0f0" exitCode=0 Jul 14 15:22:56.321072 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:56.320708 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" event={"ID":"cbb50f85-a13f-451d-a4bb-df7a9f212438","Type":"ContainerDied","Data":"c7db711a12f1cec341845aa6429f0f92d8b39e5a26bf3dd046d959bb4acad0f0"} Jul 14 15:22:56.321072 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:56.321001 2579 scope.go:117] "RemoveContainer" containerID="c7db711a12f1cec341845aa6429f0f92d8b39e5a26bf3dd046d959bb4acad0f0" Jul 14 15:22:57.325681 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:57.325645 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-zxnts" event={"ID":"cbb50f85-a13f-451d-a4bb-df7a9f212438","Type":"ContainerStarted","Data":"af587f939dcc78bee86fd1926ab9458bf27c22a81b8a1f9d68191dd80fc214b7"} Jul 14 15:22:58.277985 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:58.277945 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:58.378845 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:58.378813 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:22:59.335143 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:59.335112 2579 generic.go:356] "Generic (PLEG): container finished" podID="b0509d07-fdb6-481b-b512-43cb8056858d" containerID="7bf63d4e12b1bafa0e0ecacfc28d4aef53e09911e8956133f4684f868462e4dd" exitCode=0 Jul 14 15:22:59.335320 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:59.335195 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2zm28" event={"ID":"b0509d07-fdb6-481b-b512-43cb8056858d","Type":"ContainerDied","Data":"7bf63d4e12b1bafa0e0ecacfc28d4aef53e09911e8956133f4684f868462e4dd"} Jul 14 15:22:59.335515 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:22:59.335501 2579 scope.go:117] "RemoveContainer" containerID="7bf63d4e12b1bafa0e0ecacfc28d4aef53e09911e8956133f4684f868462e4dd" Jul 14 15:23:00.340797 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:23:00.340759 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2zm28" event={"ID":"b0509d07-fdb6-481b-b512-43cb8056858d","Type":"ContainerStarted","Data":"5f6fcedbd9f6445f3647ab066ddcba109f46b2fc1affc4772123f177da2eefe6"} Jul 14 15:23:02.360218 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:23:02.360157 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:23:02.462279 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:23:02.462241 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:23:08.373874 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:23:08.373836 2579 generic.go:356] "Generic (PLEG): container finished" podID="2d1490f8-b484-46ee-83d6-4bb7fae67fd8" containerID="2b7aa34818571c6f1bb0a865a630dd8aac6e1438d2ae30785608f612859419e2" exitCode=0 Jul 14 15:23:08.374390 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:23:08.373916 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jkhlz" event={"ID":"2d1490f8-b484-46ee-83d6-4bb7fae67fd8","Type":"ContainerDied","Data":"2b7aa34818571c6f1bb0a865a630dd8aac6e1438d2ae30785608f612859419e2"} Jul 14 15:23:08.374390 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:23:08.374360 2579 scope.go:117] "RemoveContainer" containerID="2b7aa34818571c6f1bb0a865a630dd8aac6e1438d2ae30785608f612859419e2" Jul 14 15:23:09.384042 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:23:09.384004 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-jkhlz" event={"ID":"2d1490f8-b484-46ee-83d6-4bb7fae67fd8","Type":"ContainerStarted","Data":"e8b20a7a6d612a5149a7bf92b01af1ae7cd90d99334a96d3c35b54d3285fb3fb"} Jul 14 15:23:31.267462 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:23:31.267416 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:23:43.935929 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:23:43.935886 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:23:44.147388 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:23:44.147347 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:23:54.075372 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:23:54.075274 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:23:59.171716 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:23:59.171656 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:23:59.269616 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:23:59.269580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:24:11.230819 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:11.230780 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:24:11.284741 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:11.284712 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:24:16.343457 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:16.343418 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:24:16.400623 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:16.400575 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:24:16.455657 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:16.455622 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:24:27.087980 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:27.087942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:24:27.142328 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:27.142286 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:24:28.280730 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:28.280692 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:24:28.381302 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:28.381264 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:24:38.628402 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:38.628360 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:24:55.968650 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:55.968616 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:24:56.062344 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.062313 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:24:56.533856 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.533821 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr"] Jul 14 15:24:56.534201 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.534163 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4db9b13-7375-4fbc-bd78-ed7b42e75f2c" containerName="registry" Jul 14 15:24:56.534201 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.534195 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4db9b13-7375-4fbc-bd78-ed7b42e75f2c" containerName="registry" Jul 14 15:24:56.534359 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.534270 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4db9b13-7375-4fbc-bd78-ed7b42e75f2c" containerName="registry" Jul 14 15:24:56.537142 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.537126 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr" Jul 14 15:24:56.539936 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.539916 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 15:24:56.541436 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.541418 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v8fx7\"" Jul 14 15:24:56.541436 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.541426 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 15:24:56.545031 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.545010 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr"] Jul 14 15:24:56.581134 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.581106 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx9mk\" (UniqueName: \"kubernetes.io/projected/10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f-kube-api-access-cx9mk\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr\" (UID: \"10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr" Jul 14 15:24:56.581259 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.581145 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr\" (UID: \"10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr" Jul 14 15:24:56.581259 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.581214 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr\" (UID: \"10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr" Jul 14 15:24:56.682482 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.682442 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx9mk\" (UniqueName: \"kubernetes.io/projected/10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f-kube-api-access-cx9mk\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr\" (UID: \"10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr" Jul 14 15:24:56.682632 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.682497 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr\" (UID: \"10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr" Jul 14 15:24:56.682632 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.682549 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr\" (UID: \"10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr" Jul 14 15:24:56.682889 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.682867 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr\" (UID: \"10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr" Jul 14 15:24:56.682952 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.682899 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr\" (UID: \"10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr" Jul 14 15:24:56.692299 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.692272 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx9mk\" (UniqueName: \"kubernetes.io/projected/10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f-kube-api-access-cx9mk\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr\" (UID: \"10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr" Jul 14 15:24:56.846856 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.846824 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr" Jul 14 15:24:56.976516 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:56.976439 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr"] Jul 14 15:24:56.979395 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:24:56.979367 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c5409b_15d2_4c10_b7b5_9f3e9bdcfa2f.slice/crio-b6556ba10da0059a0e60dfd6a12ba4e23a4120545f13a13f4fe73cb938c1f628 WatchSource:0}: Error finding container b6556ba10da0059a0e60dfd6a12ba4e23a4120545f13a13f4fe73cb938c1f628: Status 404 returned error can't find the container with id b6556ba10da0059a0e60dfd6a12ba4e23a4120545f13a13f4fe73cb938c1f628 Jul 14 15:24:57.291826 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:57.291721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:24:57.780894 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:24:57.780854 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr" event={"ID":"10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f","Type":"ContainerStarted","Data":"b6556ba10da0059a0e60dfd6a12ba4e23a4120545f13a13f4fe73cb938c1f628"} Jul 14 15:25:03.808509 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:03.808476 2579 generic.go:356] "Generic (PLEG): container finished" podID="10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f" containerID="30c2edd661a340e8f63ce6c2aa5ffc1a5127e8735004030e9481c73440fd0180" exitCode=0 Jul 14 15:25:03.808884 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:03.808552 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr" event={"ID":"10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f","Type":"ContainerDied","Data":"30c2edd661a340e8f63ce6c2aa5ffc1a5127e8735004030e9481c73440fd0180"} Jul 14 15:25:06.822190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:06.822128 2579 generic.go:356] "Generic (PLEG): container finished" podID="10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f" containerID="21d4e447ebcd9096082993e759312a1e8b150237e66f7075d5486292e988ba64" exitCode=0 Jul 14 15:25:06.822602 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:06.822208 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr" event={"ID":"10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f","Type":"ContainerDied","Data":"21d4e447ebcd9096082993e759312a1e8b150237e66f7075d5486292e988ba64"} Jul 14 15:25:09.554486 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:09.554437 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:25:16.864845 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:16.864807 2579 generic.go:356] "Generic (PLEG): container finished" podID="10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f" containerID="65b9e12c8227516bd7a7d4da5efd4d9217738de00ad0ac1bf0fee09d2a35e541" exitCode=0 Jul 14 15:25:16.865323 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:16.864853 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr" event={"ID":"10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f","Type":"ContainerDied","Data":"65b9e12c8227516bd7a7d4da5efd4d9217738de00ad0ac1bf0fee09d2a35e541"} Jul 14 15:25:17.992186 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:17.992125 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr" Jul 14 15:25:18.060765 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:18.060737 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f-bundle\") pod \"10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f\" (UID: \"10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f\") " Jul 14 15:25:18.060913 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:18.060783 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f-util\") pod \"10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f\" (UID: \"10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f\") " Jul 14 15:25:18.060913 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:18.060840 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cx9mk\" (UniqueName: \"kubernetes.io/projected/10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f-kube-api-access-cx9mk\") pod \"10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f\" (UID: \"10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f\") " Jul 14 15:25:18.061489 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:18.061466 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f-bundle" (OuterVolumeSpecName: "bundle") pod "10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f" (UID: "10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:25:18.063161 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:18.063127 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f-kube-api-access-cx9mk" (OuterVolumeSpecName: "kube-api-access-cx9mk") pod "10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f" (UID: "10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f"). InnerVolumeSpecName "kube-api-access-cx9mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:25:18.065425 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:18.065403 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f-util" (OuterVolumeSpecName: "util") pod "10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f" (UID: "10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:25:18.162220 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:18.162140 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f-bundle\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:25:18.162220 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:18.162164 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f-util\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:25:18.162220 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:18.162195 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx9mk\" (UniqueName: \"kubernetes.io/projected/10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f-kube-api-access-cx9mk\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:25:18.873447 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:18.873409 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr" event={"ID":"10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f","Type":"ContainerDied","Data":"b6556ba10da0059a0e60dfd6a12ba4e23a4120545f13a13f4fe73cb938c1f628"} Jul 14 15:25:18.873447 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:18.873450 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7cr" Jul 14 15:25:18.873644 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:18.873468 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6556ba10da0059a0e60dfd6a12ba4e23a4120545f13a13f4fe73cb938c1f628" Jul 14 15:25:22.545753 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:22.545639 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:25:23.307664 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:23.307624 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:25:24.504823 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.504791 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhr9"] Jul 14 15:25:24.505196 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.505077 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f" containerName="pull" Jul 14 15:25:24.505196 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.505087 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f" containerName="pull" Jul 14 15:25:24.505196 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.505099 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f" containerName="extract" Jul 14 15:25:24.505196 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.505105 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f" containerName="extract" Jul 14 15:25:24.505196 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.505111 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f" containerName="util" Jul 14 15:25:24.505196 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.505117 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f" containerName="util" Jul 14 15:25:24.505196 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.505159 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="10c5409b-15d2-4c10-b7b5-9f3e9bdcfa2f" containerName="extract" Jul 14 15:25:24.506948 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.506933 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhr9" Jul 14 15:25:24.509905 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.509880 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9dnw6\"" Jul 14 15:25:24.510089 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.510074 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 14 15:25:24.510302 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.510289 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 14 15:25:24.520229 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.520208 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhr9"] Jul 14 15:25:24.616890 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.616857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bd6471e-ec9d-4294-9ddf-fb416c1d5ed0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-qqhr9\" (UID: \"4bd6471e-ec9d-4294-9ddf-fb416c1d5ed0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhr9" Jul 14 15:25:24.617024 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.616898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqvbm\" (UniqueName: \"kubernetes.io/projected/4bd6471e-ec9d-4294-9ddf-fb416c1d5ed0-kube-api-access-qqvbm\") pod \"cert-manager-operator-controller-manager-55c444bf86-qqhr9\" (UID: \"4bd6471e-ec9d-4294-9ddf-fb416c1d5ed0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhr9" Jul 14 15:25:24.717317 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.717289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bd6471e-ec9d-4294-9ddf-fb416c1d5ed0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-qqhr9\" (UID: \"4bd6471e-ec9d-4294-9ddf-fb416c1d5ed0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhr9" Jul 14 15:25:24.717482 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.717327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqvbm\" (UniqueName: \"kubernetes.io/projected/4bd6471e-ec9d-4294-9ddf-fb416c1d5ed0-kube-api-access-qqvbm\") pod \"cert-manager-operator-controller-manager-55c444bf86-qqhr9\" (UID: \"4bd6471e-ec9d-4294-9ddf-fb416c1d5ed0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhr9" Jul 14 15:25:24.717641 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.717620 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bd6471e-ec9d-4294-9ddf-fb416c1d5ed0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-qqhr9\" (UID: \"4bd6471e-ec9d-4294-9ddf-fb416c1d5ed0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhr9" Jul 14 15:25:24.726724 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.726698 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqvbm\" (UniqueName: \"kubernetes.io/projected/4bd6471e-ec9d-4294-9ddf-fb416c1d5ed0-kube-api-access-qqvbm\") pod \"cert-manager-operator-controller-manager-55c444bf86-qqhr9\" (UID: \"4bd6471e-ec9d-4294-9ddf-fb416c1d5ed0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhr9" Jul 14 15:25:24.815898 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.815820 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhr9" Jul 14 15:25:24.967441 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:24.967406 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhr9"] Jul 14 15:25:24.973267 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:25:24.973239 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd6471e_ec9d_4294_9ddf_fb416c1d5ed0.slice/crio-036b0194c02040aa23c60a3902ec3a82925e9dc8707a89ab80e3e8605de11bac WatchSource:0}: Error finding container 036b0194c02040aa23c60a3902ec3a82925e9dc8707a89ab80e3e8605de11bac: Status 404 returned error can't find the container with id 036b0194c02040aa23c60a3902ec3a82925e9dc8707a89ab80e3e8605de11bac Jul 14 15:25:25.901024 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:25.900990 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhr9" event={"ID":"4bd6471e-ec9d-4294-9ddf-fb416c1d5ed0","Type":"ContainerStarted","Data":"036b0194c02040aa23c60a3902ec3a82925e9dc8707a89ab80e3e8605de11bac"} Jul 14 15:25:26.141481 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:26.141449 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:25:26.444600 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:26.444554 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:25:28.897823 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:28.897792 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:25:28.916264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:28.916230 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhr9" event={"ID":"4bd6471e-ec9d-4294-9ddf-fb416c1d5ed0","Type":"ContainerStarted","Data":"ec5a0dd33c736930d4007b250c3c6df3a4b57652ed91066dcb39338696579e09"} Jul 14 15:25:28.943114 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:28.943066 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhr9" podStartSLOduration=1.696837299 podStartE2EDuration="4.943051631s" totalImagesPullingTime="3.246214332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:25:28.941710022 +0000 UTC m=+269.794093180" watchObservedRunningTime="2026-07-14 15:25:28.943051631 +0000 UTC m=+269.795434797" Jul 14 15:25:28.998836 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:28.998802 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:25:29.329830 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:29.329791 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:25:29.433715 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:29.433683 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:25:29.575117 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:29.575082 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr"] Jul 14 15:25:29.578351 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:29.578334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr" Jul 14 15:25:29.581829 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:29.581774 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 15:25:29.581937 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:29.581838 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 15:25:29.583505 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:29.583481 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v8fx7\"" Jul 14 15:25:29.591378 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:29.591360 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr"] Jul 14 15:25:29.656601 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:29.656574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e590b2a9-2cfb-400e-b050-93280eb7d36c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr\" (UID: \"e590b2a9-2cfb-400e-b050-93280eb7d36c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr" Jul 14 15:25:29.656738 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:29.656629 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e590b2a9-2cfb-400e-b050-93280eb7d36c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr\" (UID: \"e590b2a9-2cfb-400e-b050-93280eb7d36c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr" Jul 14 15:25:29.656738 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:29.656657 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxhvr\" (UniqueName: \"kubernetes.io/projected/e590b2a9-2cfb-400e-b050-93280eb7d36c-kube-api-access-rxhvr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr\" (UID: \"e590b2a9-2cfb-400e-b050-93280eb7d36c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr" Jul 14 15:25:29.757034 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:29.757002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e590b2a9-2cfb-400e-b050-93280eb7d36c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr\" (UID: \"e590b2a9-2cfb-400e-b050-93280eb7d36c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr" Jul 14 15:25:29.757242 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:29.757049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e590b2a9-2cfb-400e-b050-93280eb7d36c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr\" (UID: \"e590b2a9-2cfb-400e-b050-93280eb7d36c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr" Jul 14 15:25:29.757242 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:29.757070 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxhvr\" (UniqueName: \"kubernetes.io/projected/e590b2a9-2cfb-400e-b050-93280eb7d36c-kube-api-access-rxhvr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr\" (UID: \"e590b2a9-2cfb-400e-b050-93280eb7d36c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr" Jul 14 15:25:29.757481 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:29.757459 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e590b2a9-2cfb-400e-b050-93280eb7d36c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr\" (UID: \"e590b2a9-2cfb-400e-b050-93280eb7d36c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr" Jul 14 15:25:29.757549 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:29.757500 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e590b2a9-2cfb-400e-b050-93280eb7d36c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr\" (UID: \"e590b2a9-2cfb-400e-b050-93280eb7d36c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr" Jul 14 15:25:29.767835 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:29.767814 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxhvr\" (UniqueName: \"kubernetes.io/projected/e590b2a9-2cfb-400e-b050-93280eb7d36c-kube-api-access-rxhvr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr\" (UID: \"e590b2a9-2cfb-400e-b050-93280eb7d36c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr" Jul 14 15:25:29.887685 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:29.887609 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr" Jul 14 15:25:30.020283 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:30.020253 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr"] Jul 14 15:25:30.022677 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:25:30.022649 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode590b2a9_2cfb_400e_b050_93280eb7d36c.slice/crio-7c893848216f40b81d5e43a1396d5e8d1056d0c22942f14c658bec0e6efd6118 WatchSource:0}: Error finding container 7c893848216f40b81d5e43a1396d5e8d1056d0c22942f14c658bec0e6efd6118: Status 404 returned error can't find the container with id 7c893848216f40b81d5e43a1396d5e8d1056d0c22942f14c658bec0e6efd6118 Jul 14 15:25:30.927277 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:30.927242 2579 generic.go:356] "Generic (PLEG): container finished" podID="e590b2a9-2cfb-400e-b050-93280eb7d36c" containerID="986fcb68e94cb33ecd1d525b147a6be697b798f132ac644a77390c9adab88f44" exitCode=0 Jul 14 15:25:30.927441 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:30.927325 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr" event={"ID":"e590b2a9-2cfb-400e-b050-93280eb7d36c","Type":"ContainerDied","Data":"986fcb68e94cb33ecd1d525b147a6be697b798f132ac644a77390c9adab88f44"} Jul 14 15:25:30.927441 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:30.927373 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr" event={"ID":"e590b2a9-2cfb-400e-b050-93280eb7d36c","Type":"ContainerStarted","Data":"7c893848216f40b81d5e43a1396d5e8d1056d0c22942f14c658bec0e6efd6118"} Jul 14 15:25:31.322044 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:31.322008 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:25:31.854187 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:31.854134 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:25:32.541311 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:32.541272 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:25:32.641095 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:32.641061 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:25:32.937581 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:32.937547 2579 generic.go:356] "Generic (PLEG): container finished" podID="e590b2a9-2cfb-400e-b050-93280eb7d36c" containerID="33350a89c740a443084e03aafbb6b2f30963f07d2f4524229c55bbfe5b671ca5" exitCode=0 Jul 14 15:25:32.937730 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:32.937622 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr" event={"ID":"e590b2a9-2cfb-400e-b050-93280eb7d36c","Type":"ContainerDied","Data":"33350a89c740a443084e03aafbb6b2f30963f07d2f4524229c55bbfe5b671ca5"} Jul 14 15:25:33.944461 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:33.944422 2579 generic.go:356] "Generic (PLEG): container finished" podID="e590b2a9-2cfb-400e-b050-93280eb7d36c" containerID="76fef5f2df33665481bbe171aa8230d4fc9851abbea9de39997837c8a89770e8" exitCode=0 Jul 14 15:25:33.944845 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:33.944467 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr" event={"ID":"e590b2a9-2cfb-400e-b050-93280eb7d36c","Type":"ContainerDied","Data":"76fef5f2df33665481bbe171aa8230d4fc9851abbea9de39997837c8a89770e8"} Jul 14 15:25:35.076353 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:35.076326 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr" Jul 14 15:25:35.205867 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:35.205775 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e590b2a9-2cfb-400e-b050-93280eb7d36c-bundle\") pod \"e590b2a9-2cfb-400e-b050-93280eb7d36c\" (UID: \"e590b2a9-2cfb-400e-b050-93280eb7d36c\") " Jul 14 15:25:35.205867 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:35.205825 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxhvr\" (UniqueName: \"kubernetes.io/projected/e590b2a9-2cfb-400e-b050-93280eb7d36c-kube-api-access-rxhvr\") pod \"e590b2a9-2cfb-400e-b050-93280eb7d36c\" (UID: \"e590b2a9-2cfb-400e-b050-93280eb7d36c\") " Jul 14 15:25:35.205867 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:35.205864 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e590b2a9-2cfb-400e-b050-93280eb7d36c-util\") pod \"e590b2a9-2cfb-400e-b050-93280eb7d36c\" (UID: \"e590b2a9-2cfb-400e-b050-93280eb7d36c\") " Jul 14 15:25:35.206285 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:35.206254 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e590b2a9-2cfb-400e-b050-93280eb7d36c-bundle" (OuterVolumeSpecName: "bundle") pod "e590b2a9-2cfb-400e-b050-93280eb7d36c" (UID: "e590b2a9-2cfb-400e-b050-93280eb7d36c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:25:35.208206 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:35.208166 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e590b2a9-2cfb-400e-b050-93280eb7d36c-kube-api-access-rxhvr" (OuterVolumeSpecName: "kube-api-access-rxhvr") pod "e590b2a9-2cfb-400e-b050-93280eb7d36c" (UID: "e590b2a9-2cfb-400e-b050-93280eb7d36c"). InnerVolumeSpecName "kube-api-access-rxhvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:25:35.211350 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:35.211327 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e590b2a9-2cfb-400e-b050-93280eb7d36c-util" (OuterVolumeSpecName: "util") pod "e590b2a9-2cfb-400e-b050-93280eb7d36c" (UID: "e590b2a9-2cfb-400e-b050-93280eb7d36c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:25:35.306903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:35.306866 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e590b2a9-2cfb-400e-b050-93280eb7d36c-bundle\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:25:35.306903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:35.306896 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxhvr\" (UniqueName: \"kubernetes.io/projected/e590b2a9-2cfb-400e-b050-93280eb7d36c-kube-api-access-rxhvr\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:25:35.306903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:35.306907 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e590b2a9-2cfb-400e-b050-93280eb7d36c-util\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:25:35.955746 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:35.955665 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr" Jul 14 15:25:35.955746 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:35.955685 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8mqtr" event={"ID":"e590b2a9-2cfb-400e-b050-93280eb7d36c","Type":"ContainerDied","Data":"7c893848216f40b81d5e43a1396d5e8d1056d0c22942f14c658bec0e6efd6118"} Jul 14 15:25:35.955746 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:35.955735 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c893848216f40b81d5e43a1396d5e8d1056d0c22942f14c658bec0e6efd6118" Jul 14 15:25:39.527212 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:39.527140 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:25:40.429584 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:40.429541 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:25:40.543773 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:40.543726 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:25:42.261632 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.261598 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd2t7"] Jul 14 15:25:42.261990 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.261906 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e590b2a9-2cfb-400e-b050-93280eb7d36c" containerName="pull" Jul 14 15:25:42.261990 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.261917 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e590b2a9-2cfb-400e-b050-93280eb7d36c" containerName="pull" Jul 14 15:25:42.261990 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.261923 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e590b2a9-2cfb-400e-b050-93280eb7d36c" containerName="extract" Jul 14 15:25:42.261990 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.261929 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e590b2a9-2cfb-400e-b050-93280eb7d36c" containerName="extract" Jul 14 15:25:42.261990 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.261937 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e590b2a9-2cfb-400e-b050-93280eb7d36c" containerName="util" Jul 14 15:25:42.261990 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.261942 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e590b2a9-2cfb-400e-b050-93280eb7d36c" containerName="util" Jul 14 15:25:42.262165 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.261998 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e590b2a9-2cfb-400e-b050-93280eb7d36c" containerName="extract" Jul 14 15:25:42.264943 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.264926 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd2t7" Jul 14 15:25:42.267965 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.267950 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 14 15:25:42.269651 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.269639 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 14 15:25:42.269710 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.269659 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-g268g\"" Jul 14 15:25:42.274444 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.274421 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd2t7"] Jul 14 15:25:42.365237 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.365196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzld9\" (UniqueName: \"kubernetes.io/projected/9798b5f6-2aec-4fa1-9ff3-f93ccc1493e5-kube-api-access-pzld9\") pod \"openshift-lws-operator-fd8ccff4c-gd2t7\" (UID: \"9798b5f6-2aec-4fa1-9ff3-f93ccc1493e5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd2t7" Jul 14 15:25:42.365400 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.365306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9798b5f6-2aec-4fa1-9ff3-f93ccc1493e5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gd2t7\" (UID: \"9798b5f6-2aec-4fa1-9ff3-f93ccc1493e5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd2t7" Jul 14 15:25:42.466520 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.466482 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9798b5f6-2aec-4fa1-9ff3-f93ccc1493e5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gd2t7\" (UID: \"9798b5f6-2aec-4fa1-9ff3-f93ccc1493e5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd2t7" Jul 14 15:25:42.466701 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.466536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzld9\" (UniqueName: \"kubernetes.io/projected/9798b5f6-2aec-4fa1-9ff3-f93ccc1493e5-kube-api-access-pzld9\") pod \"openshift-lws-operator-fd8ccff4c-gd2t7\" (UID: \"9798b5f6-2aec-4fa1-9ff3-f93ccc1493e5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd2t7" Jul 14 15:25:42.466909 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.466886 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9798b5f6-2aec-4fa1-9ff3-f93ccc1493e5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gd2t7\" (UID: \"9798b5f6-2aec-4fa1-9ff3-f93ccc1493e5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd2t7" Jul 14 15:25:42.475787 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.475766 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzld9\" (UniqueName: \"kubernetes.io/projected/9798b5f6-2aec-4fa1-9ff3-f93ccc1493e5-kube-api-access-pzld9\") pod \"openshift-lws-operator-fd8ccff4c-gd2t7\" (UID: \"9798b5f6-2aec-4fa1-9ff3-f93ccc1493e5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd2t7" Jul 14 15:25:42.585902 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.585871 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd2t7" Jul 14 15:25:42.718530 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.718503 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd2t7"] Jul 14 15:25:42.720289 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:25:42.720260 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9798b5f6_2aec_4fa1_9ff3_f93ccc1493e5.slice/crio-a1c5d386807fc15ee13772382920c5159cb9ab8d68c81688c18329f82c29461f WatchSource:0}: Error finding container a1c5d386807fc15ee13772382920c5159cb9ab8d68c81688c18329f82c29461f: Status 404 returned error can't find the container with id a1c5d386807fc15ee13772382920c5159cb9ab8d68c81688c18329f82c29461f Jul 14 15:25:42.986973 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:42.986888 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd2t7" event={"ID":"9798b5f6-2aec-4fa1-9ff3-f93ccc1493e5","Type":"ContainerStarted","Data":"a1c5d386807fc15ee13772382920c5159cb9ab8d68c81688c18329f82c29461f"} Jul 14 15:25:44.998050 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:44.997951 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd2t7" event={"ID":"9798b5f6-2aec-4fa1-9ff3-f93ccc1493e5","Type":"ContainerStarted","Data":"93790a46226bf139e61c121666633fff91aca7b7b8ab0d9358b4bd3d01edbcbd"} Jul 14 15:25:45.016348 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:45.016290 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gd2t7" podStartSLOduration=1.148433505 podStartE2EDuration="3.016270131s" totalImagesPullingTime="1.867836626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:25:45.014082264 +0000 UTC m=+285.866465430" watchObservedRunningTime="2026-07-14 15:25:45.016270131 +0000 UTC m=+285.868653297" Jul 14 15:25:45.759691 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:45.759648 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:25:45.862787 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:45.862753 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:25:47.650458 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:47.650424 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf"] Jul 14 15:25:47.665346 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:47.665312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:25:47.672191 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:47.672148 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf"] Jul 14 15:25:47.672293 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:47.672279 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf" Jul 14 15:25:47.675235 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:47.675214 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 15:25:47.675359 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:47.675339 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v8fx7\"" Jul 14 15:25:47.675359 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:47.675354 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 15:25:47.707979 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:47.707954 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7clc7\" (UniqueName: \"kubernetes.io/projected/0287e698-5f1c-48e1-b765-d56be2db7897-kube-api-access-7clc7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf\" (UID: \"0287e698-5f1c-48e1-b765-d56be2db7897\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf" Jul 14 15:25:47.708076 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:47.707987 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0287e698-5f1c-48e1-b765-d56be2db7897-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf\" (UID: \"0287e698-5f1c-48e1-b765-d56be2db7897\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf" Jul 14 15:25:47.708118 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:47.708082 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0287e698-5f1c-48e1-b765-d56be2db7897-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf\" (UID: \"0287e698-5f1c-48e1-b765-d56be2db7897\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf" Jul 14 15:25:47.808593 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:47.808559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0287e698-5f1c-48e1-b765-d56be2db7897-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf\" (UID: \"0287e698-5f1c-48e1-b765-d56be2db7897\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf" Jul 14 15:25:47.808739 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:47.808625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0287e698-5f1c-48e1-b765-d56be2db7897-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf\" (UID: \"0287e698-5f1c-48e1-b765-d56be2db7897\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf" Jul 14 15:25:47.808739 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:47.808668 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7clc7\" (UniqueName: \"kubernetes.io/projected/0287e698-5f1c-48e1-b765-d56be2db7897-kube-api-access-7clc7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf\" (UID: \"0287e698-5f1c-48e1-b765-d56be2db7897\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf" Jul 14 15:25:47.808971 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:47.808951 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0287e698-5f1c-48e1-b765-d56be2db7897-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf\" (UID: \"0287e698-5f1c-48e1-b765-d56be2db7897\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf" Jul 14 15:25:47.809028 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:47.808978 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0287e698-5f1c-48e1-b765-d56be2db7897-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf\" (UID: \"0287e698-5f1c-48e1-b765-d56be2db7897\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf" Jul 14 15:25:47.818118 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:47.818095 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7clc7\" (UniqueName: \"kubernetes.io/projected/0287e698-5f1c-48e1-b765-d56be2db7897-kube-api-access-7clc7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf\" (UID: \"0287e698-5f1c-48e1-b765-d56be2db7897\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf" Jul 14 15:25:47.981547 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:47.981465 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf" Jul 14 15:25:48.136935 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:48.136895 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf"] Jul 14 15:25:48.139635 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:25:48.139607 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0287e698_5f1c_48e1_b765_d56be2db7897.slice/crio-caa138394d8fc237d9f28ba344f1a90f119f2e4258ff77513f518097960a8b2e WatchSource:0}: Error finding container caa138394d8fc237d9f28ba344f1a90f119f2e4258ff77513f518097960a8b2e: Status 404 returned error can't find the container with id caa138394d8fc237d9f28ba344f1a90f119f2e4258ff77513f518097960a8b2e Jul 14 15:25:49.015705 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:49.015661 2579 generic.go:356] "Generic (PLEG): container finished" podID="0287e698-5f1c-48e1-b765-d56be2db7897" containerID="54bb8c6d578112ade859d40279a86bd916ef7c76602dc35f3b91415d8c3b0ada" exitCode=0 Jul 14 15:25:49.016053 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:49.015752 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf" event={"ID":"0287e698-5f1c-48e1-b765-d56be2db7897","Type":"ContainerDied","Data":"54bb8c6d578112ade859d40279a86bd916ef7c76602dc35f3b91415d8c3b0ada"} Jul 14 15:25:49.016053 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:49.015839 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf" event={"ID":"0287e698-5f1c-48e1-b765-d56be2db7897","Type":"ContainerStarted","Data":"caa138394d8fc237d9f28ba344f1a90f119f2e4258ff77513f518097960a8b2e"} Jul 14 15:25:51.025992 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:51.025951 2579 generic.go:356] "Generic (PLEG): container finished" podID="0287e698-5f1c-48e1-b765-d56be2db7897" containerID="046b42f9f6eff9459ea19221b88bc6043f05b456368507cc2b6da1e360a68746" exitCode=0 Jul 14 15:25:51.026452 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:51.026044 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf" event={"ID":"0287e698-5f1c-48e1-b765-d56be2db7897","Type":"ContainerDied","Data":"046b42f9f6eff9459ea19221b88bc6043f05b456368507cc2b6da1e360a68746"} Jul 14 15:25:52.033101 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:52.033070 2579 generic.go:356] "Generic (PLEG): container finished" podID="0287e698-5f1c-48e1-b765-d56be2db7897" containerID="338659020d05f53099f7e52708909908ca25050711277021f6d07da550e0b233" exitCode=0 Jul 14 15:25:52.033484 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:52.033149 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf" event={"ID":"0287e698-5f1c-48e1-b765-d56be2db7897","Type":"ContainerDied","Data":"338659020d05f53099f7e52708909908ca25050711277021f6d07da550e0b233"} Jul 14 15:25:53.158875 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:53.158852 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf" Jul 14 15:25:53.253552 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:53.253524 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0287e698-5f1c-48e1-b765-d56be2db7897-util\") pod \"0287e698-5f1c-48e1-b765-d56be2db7897\" (UID: \"0287e698-5f1c-48e1-b765-d56be2db7897\") " Jul 14 15:25:53.253835 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:53.253594 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0287e698-5f1c-48e1-b765-d56be2db7897-bundle\") pod \"0287e698-5f1c-48e1-b765-d56be2db7897\" (UID: \"0287e698-5f1c-48e1-b765-d56be2db7897\") " Jul 14 15:25:53.253835 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:53.253629 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7clc7\" (UniqueName: \"kubernetes.io/projected/0287e698-5f1c-48e1-b765-d56be2db7897-kube-api-access-7clc7\") pod \"0287e698-5f1c-48e1-b765-d56be2db7897\" (UID: \"0287e698-5f1c-48e1-b765-d56be2db7897\") " Jul 14 15:25:53.254471 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:53.254433 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0287e698-5f1c-48e1-b765-d56be2db7897-bundle" (OuterVolumeSpecName: "bundle") pod "0287e698-5f1c-48e1-b765-d56be2db7897" (UID: "0287e698-5f1c-48e1-b765-d56be2db7897"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:25:53.255734 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:53.255711 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0287e698-5f1c-48e1-b765-d56be2db7897-kube-api-access-7clc7" (OuterVolumeSpecName: "kube-api-access-7clc7") pod "0287e698-5f1c-48e1-b765-d56be2db7897" (UID: "0287e698-5f1c-48e1-b765-d56be2db7897"). InnerVolumeSpecName "kube-api-access-7clc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:25:53.259105 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:53.259075 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0287e698-5f1c-48e1-b765-d56be2db7897-util" (OuterVolumeSpecName: "util") pod "0287e698-5f1c-48e1-b765-d56be2db7897" (UID: "0287e698-5f1c-48e1-b765-d56be2db7897"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:25:53.354873 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:53.354843 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0287e698-5f1c-48e1-b765-d56be2db7897-bundle\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:25:53.354873 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:53.354872 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7clc7\" (UniqueName: \"kubernetes.io/projected/0287e698-5f1c-48e1-b765-d56be2db7897-kube-api-access-7clc7\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:25:53.355052 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:53.354884 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0287e698-5f1c-48e1-b765-d56be2db7897-util\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:25:54.047972 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:54.047895 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf" Jul 14 15:25:54.048123 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:54.047893 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqhqhf" event={"ID":"0287e698-5f1c-48e1-b765-d56be2db7897","Type":"ContainerDied","Data":"caa138394d8fc237d9f28ba344f1a90f119f2e4258ff77513f518097960a8b2e"} Jul 14 15:25:54.048123 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:54.048030 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caa138394d8fc237d9f28ba344f1a90f119f2e4258ff77513f518097960a8b2e" Jul 14 15:25:59.577556 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:59.577530 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 15:25:59.582988 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:59.582966 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 15:25:59.589497 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:59.589460 2579 kubelet.go:1648] "Image garbage collection succeeded" Jul 14 15:25:59.706251 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:59.706074 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:25:59.758438 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:25:59.758402 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:03.602070 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:03.602036 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:04.750670 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.750635 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b"] Jul 14 15:26:04.751023 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.750913 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0287e698-5f1c-48e1-b765-d56be2db7897" containerName="extract" Jul 14 15:26:04.751023 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.750923 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0287e698-5f1c-48e1-b765-d56be2db7897" containerName="extract" Jul 14 15:26:04.751023 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.750938 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0287e698-5f1c-48e1-b765-d56be2db7897" containerName="util" Jul 14 15:26:04.751023 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.750943 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0287e698-5f1c-48e1-b765-d56be2db7897" containerName="util" Jul 14 15:26:04.751023 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.750955 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0287e698-5f1c-48e1-b765-d56be2db7897" containerName="pull" Jul 14 15:26:04.751023 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.750960 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0287e698-5f1c-48e1-b765-d56be2db7897" containerName="pull" Jul 14 15:26:04.751226 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.751028 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0287e698-5f1c-48e1-b765-d56be2db7897" containerName="extract" Jul 14 15:26:04.754973 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.754952 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" Jul 14 15:26:04.758494 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.758473 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 14 15:26:04.758629 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.758507 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 14 15:26:04.758629 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.758538 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 14 15:26:04.758629 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.758544 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-765p8\"" Jul 14 15:26:04.758629 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.758616 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 14 15:26:04.770457 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.770438 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b"] Jul 14 15:26:04.849490 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.849459 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/027d63ac-3923-4ea6-bd02-c2d7f45de26d-webhook-cert\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-62w2b\" (UID: \"027d63ac-3923-4ea6-bd02-c2d7f45de26d\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" Jul 14 15:26:04.849657 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.849507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9wqq\" (UniqueName: \"kubernetes.io/projected/027d63ac-3923-4ea6-bd02-c2d7f45de26d-kube-api-access-p9wqq\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-62w2b\" (UID: \"027d63ac-3923-4ea6-bd02-c2d7f45de26d\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" Jul 14 15:26:04.849657 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.849582 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/027d63ac-3923-4ea6-bd02-c2d7f45de26d-tmp\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-62w2b\" (UID: \"027d63ac-3923-4ea6-bd02-c2d7f45de26d\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" Jul 14 15:26:04.849752 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.849667 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/027d63ac-3923-4ea6-bd02-c2d7f45de26d-manifests\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-62w2b\" (UID: \"027d63ac-3923-4ea6-bd02-c2d7f45de26d\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" Jul 14 15:26:04.849752 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.849697 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/027d63ac-3923-4ea6-bd02-c2d7f45de26d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-62w2b\" (UID: \"027d63ac-3923-4ea6-bd02-c2d7f45de26d\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" Jul 14 15:26:04.950648 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.950618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/027d63ac-3923-4ea6-bd02-c2d7f45de26d-manifests\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-62w2b\" (UID: \"027d63ac-3923-4ea6-bd02-c2d7f45de26d\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" Jul 14 15:26:04.950833 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.950659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/027d63ac-3923-4ea6-bd02-c2d7f45de26d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-62w2b\" (UID: \"027d63ac-3923-4ea6-bd02-c2d7f45de26d\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" Jul 14 15:26:04.950833 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.950707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/027d63ac-3923-4ea6-bd02-c2d7f45de26d-webhook-cert\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-62w2b\" (UID: \"027d63ac-3923-4ea6-bd02-c2d7f45de26d\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" Jul 14 15:26:04.950833 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.950742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9wqq\" (UniqueName: \"kubernetes.io/projected/027d63ac-3923-4ea6-bd02-c2d7f45de26d-kube-api-access-p9wqq\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-62w2b\" (UID: \"027d63ac-3923-4ea6-bd02-c2d7f45de26d\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" Jul 14 15:26:04.950833 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.950784 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/027d63ac-3923-4ea6-bd02-c2d7f45de26d-tmp\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-62w2b\" (UID: \"027d63ac-3923-4ea6-bd02-c2d7f45de26d\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" Jul 14 15:26:04.951082 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.951059 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/027d63ac-3923-4ea6-bd02-c2d7f45de26d-manifests\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-62w2b\" (UID: \"027d63ac-3923-4ea6-bd02-c2d7f45de26d\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" Jul 14 15:26:04.953064 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.953032 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/027d63ac-3923-4ea6-bd02-c2d7f45de26d-tmp\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-62w2b\" (UID: \"027d63ac-3923-4ea6-bd02-c2d7f45de26d\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" Jul 14 15:26:04.953278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.953262 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/027d63ac-3923-4ea6-bd02-c2d7f45de26d-webhook-cert\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-62w2b\" (UID: \"027d63ac-3923-4ea6-bd02-c2d7f45de26d\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" Jul 14 15:26:04.953397 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.953377 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/027d63ac-3923-4ea6-bd02-c2d7f45de26d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-62w2b\" (UID: \"027d63ac-3923-4ea6-bd02-c2d7f45de26d\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" Jul 14 15:26:04.960680 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:04.960654 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9wqq\" (UniqueName: \"kubernetes.io/projected/027d63ac-3923-4ea6-bd02-c2d7f45de26d-kube-api-access-p9wqq\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-62w2b\" (UID: \"027d63ac-3923-4ea6-bd02-c2d7f45de26d\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" Jul 14 15:26:05.064839 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:05.064756 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" Jul 14 15:26:05.205458 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:05.205433 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b"] Jul 14 15:26:05.207495 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:26:05.207457 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027d63ac_3923_4ea6_bd02_c2d7f45de26d.slice/crio-10da32ce6d2e968b93dcbc72702020041a0815a8b1d33482766c913c139c4202 WatchSource:0}: Error finding container 10da32ce6d2e968b93dcbc72702020041a0815a8b1d33482766c913c139c4202: Status 404 returned error can't find the container with id 10da32ce6d2e968b93dcbc72702020041a0815a8b1d33482766c913c139c4202 Jul 14 15:26:05.209294 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:05.209274 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 15:26:05.509609 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:05.509576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:05.613698 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:05.613663 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:06.098738 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:06.098705 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" event={"ID":"027d63ac-3923-4ea6-bd02-c2d7f45de26d","Type":"ContainerStarted","Data":"10da32ce6d2e968b93dcbc72702020041a0815a8b1d33482766c913c139c4202"} Jul 14 15:26:07.926804 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:07.926762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:10.122544 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:10.122508 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" event={"ID":"027d63ac-3923-4ea6-bd02-c2d7f45de26d","Type":"ContainerStarted","Data":"354f3376d4430d935204783d8f9279ce5b2ecc06f4b94698bd703e1bec32dc2c"} Jul 14 15:26:11.128059 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:11.128023 2579 generic.go:356] "Generic (PLEG): container finished" podID="027d63ac-3923-4ea6-bd02-c2d7f45de26d" containerID="354f3376d4430d935204783d8f9279ce5b2ecc06f4b94698bd703e1bec32dc2c" exitCode=0 Jul 14 15:26:11.128449 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:11.128084 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" event={"ID":"027d63ac-3923-4ea6-bd02-c2d7f45de26d","Type":"ContainerDied","Data":"354f3376d4430d935204783d8f9279ce5b2ecc06f4b94698bd703e1bec32dc2c"} Jul 14 15:26:11.450371 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:11.450267 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:14.144954 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:14.144908 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" event={"ID":"027d63ac-3923-4ea6-bd02-c2d7f45de26d","Type":"ContainerStarted","Data":"1d5a8ff3c59e588aa4d13a919b323590028824d51c922c0d06590d9b0b0620ed"} Jul 14 15:26:14.145405 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:14.145024 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" Jul 14 15:26:14.171112 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:14.171065 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" podStartSLOduration=2.6852078280000002 podStartE2EDuration="10.171051556s" totalImagesPullingTime="6.394131788s" totalInitContainerRuntime="1.09171194s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:26:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:26:14.168638174 +0000 UTC m=+315.021021338" watchObservedRunningTime="2026-07-14 15:26:14.171051556 +0000 UTC m=+315.023434719" Jul 14 15:26:17.515414 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:17.515378 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:17.620121 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:17.620082 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:23.488847 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:23.488814 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:23.588992 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:23.588926 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:25.152728 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:25.152697 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-62w2b" Jul 14 15:26:26.948244 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:26.948206 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:27.044115 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.044068 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:27.223503 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.223417 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:27.356693 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.356661 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7f5f96f494-p85qh"] Jul 14 15:26:27.360209 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.360189 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f5f96f494-p85qh" Jul 14 15:26:27.363024 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.362994 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 14 15:26:27.363134 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.363063 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 14 15:26:27.363134 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.363098 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6j5mf\"" Jul 14 15:26:27.369265 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.369245 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f5f96f494-p85qh"] Jul 14 15:26:27.441451 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.441411 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v97q\" (UniqueName: \"kubernetes.io/projected/dab2d01e-5dd1-4a4c-8054-fa921782c9ea-kube-api-access-2v97q\") pod \"kube-auth-proxy-7f5f96f494-p85qh\" (UID: \"dab2d01e-5dd1-4a4c-8054-fa921782c9ea\") " pod="openshift-ingress/kube-auth-proxy-7f5f96f494-p85qh" Jul 14 15:26:27.441622 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.441465 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dab2d01e-5dd1-4a4c-8054-fa921782c9ea-tmp\") pod \"kube-auth-proxy-7f5f96f494-p85qh\" (UID: \"dab2d01e-5dd1-4a4c-8054-fa921782c9ea\") " pod="openshift-ingress/kube-auth-proxy-7f5f96f494-p85qh" Jul 14 15:26:27.441622 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.441509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dab2d01e-5dd1-4a4c-8054-fa921782c9ea-tls-certs\") pod \"kube-auth-proxy-7f5f96f494-p85qh\" (UID: \"dab2d01e-5dd1-4a4c-8054-fa921782c9ea\") " pod="openshift-ingress/kube-auth-proxy-7f5f96f494-p85qh" Jul 14 15:26:27.542538 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.542458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dab2d01e-5dd1-4a4c-8054-fa921782c9ea-tls-certs\") pod \"kube-auth-proxy-7f5f96f494-p85qh\" (UID: \"dab2d01e-5dd1-4a4c-8054-fa921782c9ea\") " pod="openshift-ingress/kube-auth-proxy-7f5f96f494-p85qh" Jul 14 15:26:27.542686 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.542547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v97q\" (UniqueName: \"kubernetes.io/projected/dab2d01e-5dd1-4a4c-8054-fa921782c9ea-kube-api-access-2v97q\") pod \"kube-auth-proxy-7f5f96f494-p85qh\" (UID: \"dab2d01e-5dd1-4a4c-8054-fa921782c9ea\") " pod="openshift-ingress/kube-auth-proxy-7f5f96f494-p85qh" Jul 14 15:26:27.542686 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.542584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dab2d01e-5dd1-4a4c-8054-fa921782c9ea-tmp\") pod \"kube-auth-proxy-7f5f96f494-p85qh\" (UID: \"dab2d01e-5dd1-4a4c-8054-fa921782c9ea\") " pod="openshift-ingress/kube-auth-proxy-7f5f96f494-p85qh" Jul 14 15:26:27.545027 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.545003 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dab2d01e-5dd1-4a4c-8054-fa921782c9ea-tmp\") pod \"kube-auth-proxy-7f5f96f494-p85qh\" (UID: \"dab2d01e-5dd1-4a4c-8054-fa921782c9ea\") " pod="openshift-ingress/kube-auth-proxy-7f5f96f494-p85qh" Jul 14 15:26:27.545354 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.545334 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dab2d01e-5dd1-4a4c-8054-fa921782c9ea-tls-certs\") pod \"kube-auth-proxy-7f5f96f494-p85qh\" (UID: \"dab2d01e-5dd1-4a4c-8054-fa921782c9ea\") " pod="openshift-ingress/kube-auth-proxy-7f5f96f494-p85qh" Jul 14 15:26:27.554692 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.554670 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v97q\" (UniqueName: \"kubernetes.io/projected/dab2d01e-5dd1-4a4c-8054-fa921782c9ea-kube-api-access-2v97q\") pod \"kube-auth-proxy-7f5f96f494-p85qh\" (UID: \"dab2d01e-5dd1-4a4c-8054-fa921782c9ea\") " pod="openshift-ingress/kube-auth-proxy-7f5f96f494-p85qh" Jul 14 15:26:27.579283 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.579259 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6tgk4"] Jul 14 15:26:27.582595 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.582580 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-6tgk4" Jul 14 15:26:27.586005 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.585990 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 14 15:26:27.586254 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.586238 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-scvzz\"" Jul 14 15:26:27.594744 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.594722 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6tgk4"] Jul 14 15:26:27.670073 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.670024 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f5f96f494-p85qh" Jul 14 15:26:27.745367 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.745329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzx6c\" (UniqueName: \"kubernetes.io/projected/dc512517-286f-46e6-b332-34c74d649ed1-kube-api-access-fzx6c\") pod \"odh-model-controller-7998c67dc4-6tgk4\" (UID: \"dc512517-286f-46e6-b332-34c74d649ed1\") " pod="opendatahub/odh-model-controller-7998c67dc4-6tgk4" Jul 14 15:26:27.745676 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.745655 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc512517-286f-46e6-b332-34c74d649ed1-cert\") pod \"odh-model-controller-7998c67dc4-6tgk4\" (UID: \"dc512517-286f-46e6-b332-34c74d649ed1\") " pod="opendatahub/odh-model-controller-7998c67dc4-6tgk4" Jul 14 15:26:27.809405 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.809382 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f5f96f494-p85qh"] Jul 14 15:26:27.811064 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:26:27.811036 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab2d01e_5dd1_4a4c_8054_fa921782c9ea.slice/crio-8f2c1665f8bff8eb839db5b068509f33c19cd3c8f07aec8342ad9aca351083c5 WatchSource:0}: Error finding container 8f2c1665f8bff8eb839db5b068509f33c19cd3c8f07aec8342ad9aca351083c5: Status 404 returned error can't find the container with id 8f2c1665f8bff8eb839db5b068509f33c19cd3c8f07aec8342ad9aca351083c5 Jul 14 15:26:27.846336 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.846311 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzx6c\" (UniqueName: \"kubernetes.io/projected/dc512517-286f-46e6-b332-34c74d649ed1-kube-api-access-fzx6c\") pod \"odh-model-controller-7998c67dc4-6tgk4\" (UID: \"dc512517-286f-46e6-b332-34c74d649ed1\") " pod="opendatahub/odh-model-controller-7998c67dc4-6tgk4" Jul 14 15:26:27.846489 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.846418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc512517-286f-46e6-b332-34c74d649ed1-cert\") pod \"odh-model-controller-7998c67dc4-6tgk4\" (UID: \"dc512517-286f-46e6-b332-34c74d649ed1\") " pod="opendatahub/odh-model-controller-7998c67dc4-6tgk4" Jul 14 15:26:27.848841 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.848817 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc512517-286f-46e6-b332-34c74d649ed1-cert\") pod \"odh-model-controller-7998c67dc4-6tgk4\" (UID: \"dc512517-286f-46e6-b332-34c74d649ed1\") " pod="opendatahub/odh-model-controller-7998c67dc4-6tgk4" Jul 14 15:26:27.855563 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.855545 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzx6c\" (UniqueName: \"kubernetes.io/projected/dc512517-286f-46e6-b332-34c74d649ed1-kube-api-access-fzx6c\") pod \"odh-model-controller-7998c67dc4-6tgk4\" (UID: \"dc512517-286f-46e6-b332-34c74d649ed1\") " pod="opendatahub/odh-model-controller-7998c67dc4-6tgk4" Jul 14 15:26:27.892828 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:27.892809 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-6tgk4" Jul 14 15:26:28.029368 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.029341 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6tgk4"] Jul 14 15:26:28.031128 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:26:28.031100 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc512517_286f_46e6_b332_34c74d649ed1.slice/crio-c11cdbe889cabfef776d9901d78258d62ec430642dd8ac6f9e5eee9fd5b28928 WatchSource:0}: Error finding container c11cdbe889cabfef776d9901d78258d62ec430642dd8ac6f9e5eee9fd5b28928: Status 404 returned error can't find the container with id c11cdbe889cabfef776d9901d78258d62ec430642dd8ac6f9e5eee9fd5b28928 Jul 14 15:26:28.208960 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.208922 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f5f96f494-p85qh" event={"ID":"dab2d01e-5dd1-4a4c-8054-fa921782c9ea","Type":"ContainerStarted","Data":"8f2c1665f8bff8eb839db5b068509f33c19cd3c8f07aec8342ad9aca351083c5"} Jul 14 15:26:28.209929 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.209904 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-6tgk4" event={"ID":"dc512517-286f-46e6-b332-34c74d649ed1","Type":"ContainerStarted","Data":"c11cdbe889cabfef776d9901d78258d62ec430642dd8ac6f9e5eee9fd5b28928"} Jul 14 15:26:28.393199 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.393155 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ddsx7"] Jul 14 15:26:28.397655 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.397635 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ddsx7" Jul 14 15:26:28.400741 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.400719 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 14 15:26:28.400852 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.400720 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-lmmst\"" Jul 14 15:26:28.411883 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.409006 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ddsx7"] Jul 14 15:26:28.495411 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.495325 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n"] Jul 14 15:26:28.498821 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.498797 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.501876 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.501857 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 14 15:26:28.502033 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.501857 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 14 15:26:28.502112 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.501858 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 14 15:26:28.502232 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.501882 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 14 15:26:28.502300 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.501992 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rvlt8\"" Jul 14 15:26:28.509247 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.509223 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n"] Jul 14 15:26:28.551457 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.551429 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chswc\" (UniqueName: \"kubernetes.io/projected/6f993324-9ef2-4b07-8238-4716dc18ea75-kube-api-access-chswc\") pod \"model-serving-api-54d89bd5fc-ddsx7\" (UID: \"6f993324-9ef2-4b07-8238-4716dc18ea75\") " pod="opendatahub/model-serving-api-54d89bd5fc-ddsx7" Jul 14 15:26:28.551608 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.551530 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f993324-9ef2-4b07-8238-4716dc18ea75-tls-certs\") pod \"model-serving-api-54d89bd5fc-ddsx7\" (UID: \"6f993324-9ef2-4b07-8238-4716dc18ea75\") " pod="opendatahub/model-serving-api-54d89bd5fc-ddsx7" Jul 14 15:26:28.652222 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.652158 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f993324-9ef2-4b07-8238-4716dc18ea75-tls-certs\") pod \"model-serving-api-54d89bd5fc-ddsx7\" (UID: \"6f993324-9ef2-4b07-8238-4716dc18ea75\") " pod="opendatahub/model-serving-api-54d89bd5fc-ddsx7" Jul 14 15:26:28.652402 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.652245 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.652402 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.652275 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d58cdb-8e6f-4a39-acb2-15967443501f-local-certs\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.652402 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.652305 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-token\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.652402 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.652337 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.652402 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:26:28.652360 2579 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 14 15:26:28.652626 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.652364 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7d58cdb-8e6f-4a39-acb2-15967443501f-cacerts\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.652626 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:26:28.652438 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f993324-9ef2-4b07-8238-4716dc18ea75-tls-certs podName:6f993324-9ef2-4b07-8238-4716dc18ea75 nodeName:}" failed. No retries permitted until 2026-07-14 15:26:29.152415996 +0000 UTC m=+330.004799158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6f993324-9ef2-4b07-8238-4716dc18ea75-tls-certs") pod "model-serving-api-54d89bd5fc-ddsx7" (UID: "6f993324-9ef2-4b07-8238-4716dc18ea75") : secret "model-serving-api-tls" not found Jul 14 15:26:28.652626 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.652478 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r5lr\" (UniqueName: \"kubernetes.io/projected/e7d58cdb-8e6f-4a39-acb2-15967443501f-kube-api-access-5r5lr\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.652626 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.652536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chswc\" (UniqueName: \"kubernetes.io/projected/6f993324-9ef2-4b07-8238-4716dc18ea75-kube-api-access-chswc\") pod \"model-serving-api-54d89bd5fc-ddsx7\" (UID: \"6f993324-9ef2-4b07-8238-4716dc18ea75\") " pod="opendatahub/model-serving-api-54d89bd5fc-ddsx7" Jul 14 15:26:28.652626 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.652578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.664383 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.664359 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chswc\" (UniqueName: \"kubernetes.io/projected/6f993324-9ef2-4b07-8238-4716dc18ea75-kube-api-access-chswc\") pod \"model-serving-api-54d89bd5fc-ddsx7\" (UID: \"6f993324-9ef2-4b07-8238-4716dc18ea75\") " pod="opendatahub/model-serving-api-54d89bd5fc-ddsx7" Jul 14 15:26:28.754145 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.753744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.754145 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.753782 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d58cdb-8e6f-4a39-acb2-15967443501f-local-certs\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.754145 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.753807 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-token\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.754145 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.753834 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.754145 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.753864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7d58cdb-8e6f-4a39-acb2-15967443501f-cacerts\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.754145 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.753885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r5lr\" (UniqueName: \"kubernetes.io/projected/e7d58cdb-8e6f-4a39-acb2-15967443501f-kube-api-access-5r5lr\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.754145 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.753931 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.754754 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.754722 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.757490 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.756820 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d58cdb-8e6f-4a39-acb2-15967443501f-local-certs\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.757490 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.756975 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.758516 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.758040 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7d58cdb-8e6f-4a39-acb2-15967443501f-cacerts\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.758516 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.758364 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.765783 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.765741 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-token\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.767018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.766813 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r5lr\" (UniqueName: \"kubernetes.io/projected/e7d58cdb-8e6f-4a39-acb2-15967443501f-kube-api-access-5r5lr\") pod \"istiod-openshift-gateway-7574459656-zxn6n\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.810710 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.810647 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:28.896668 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.896526 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:28.977403 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.977371 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n"] Jul 14 15:26:28.980300 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:26:28.980058 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d58cdb_8e6f_4a39_acb2_15967443501f.slice/crio-ca83b6347ba0b7153b8540140471e91791303db5867d50655f70fccbc535d73e WatchSource:0}: Error finding container ca83b6347ba0b7153b8540140471e91791303db5867d50655f70fccbc535d73e: Status 404 returned error can't find the container with id ca83b6347ba0b7153b8540140471e91791303db5867d50655f70fccbc535d73e Jul 14 15:26:28.980976 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:28.980907 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:29.158893 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:29.158858 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f993324-9ef2-4b07-8238-4716dc18ea75-tls-certs\") pod \"model-serving-api-54d89bd5fc-ddsx7\" (UID: \"6f993324-9ef2-4b07-8238-4716dc18ea75\") " pod="opendatahub/model-serving-api-54d89bd5fc-ddsx7" Jul 14 15:26:29.162646 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:29.162620 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f993324-9ef2-4b07-8238-4716dc18ea75-tls-certs\") pod \"model-serving-api-54d89bd5fc-ddsx7\" (UID: \"6f993324-9ef2-4b07-8238-4716dc18ea75\") " pod="opendatahub/model-serving-api-54d89bd5fc-ddsx7" Jul 14 15:26:29.217743 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:29.217707 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" event={"ID":"e7d58cdb-8e6f-4a39-acb2-15967443501f","Type":"ContainerStarted","Data":"ca83b6347ba0b7153b8540140471e91791303db5867d50655f70fccbc535d73e"} Jul 14 15:26:29.315462 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:29.315431 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ddsx7" Jul 14 15:26:29.426869 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:29.426734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:29.476285 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:29.476256 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ddsx7"] Jul 14 15:26:29.478021 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:26:29.477983 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f993324_9ef2_4b07_8238_4716dc18ea75.slice/crio-c908e63d8d14596d793dcb93e05006b8cd07797559b63c24f32d4d9dfadc7974 WatchSource:0}: Error finding container c908e63d8d14596d793dcb93e05006b8cd07797559b63c24f32d4d9dfadc7974: Status 404 returned error can't find the container with id c908e63d8d14596d793dcb93e05006b8cd07797559b63c24f32d4d9dfadc7974 Jul 14 15:26:30.227357 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:30.227310 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ddsx7" event={"ID":"6f993324-9ef2-4b07-8238-4716dc18ea75","Type":"ContainerStarted","Data":"c908e63d8d14596d793dcb93e05006b8cd07797559b63c24f32d4d9dfadc7974"} Jul 14 15:26:33.509926 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:33.509891 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 15:26:33.510362 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:33.509969 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 15:26:34.252081 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:34.252043 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f5f96f494-p85qh" event={"ID":"dab2d01e-5dd1-4a4c-8054-fa921782c9ea","Type":"ContainerStarted","Data":"9b85c956b0f32e72271a6b9a82234d08ee989f5d9593c9bd28a75e9296d2192c"} Jul 14 15:26:34.254047 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:34.254007 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" event={"ID":"e7d58cdb-8e6f-4a39-acb2-15967443501f","Type":"ContainerStarted","Data":"1ee837a1d9b760c0e77deaed23ba8f77775b47cd0da3d944b3e7d46a2058e6fb"} Jul 14 15:26:34.254262 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:34.254230 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:34.256025 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:34.255997 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ddsx7" event={"ID":"6f993324-9ef2-4b07-8238-4716dc18ea75","Type":"ContainerStarted","Data":"d3726bed8b32ace599a523e8cdad32513b500d99fb414343c05731e2b4153c7c"} Jul 14 15:26:34.256137 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:34.256071 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-ddsx7" Jul 14 15:26:34.256704 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:34.256683 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-zxn6n container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 15:26:34.256790 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:34.256725 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" podUID="e7d58cdb-8e6f-4a39-acb2-15967443501f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 15:26:34.257928 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:34.257908 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-6tgk4" event={"ID":"dc512517-286f-46e6-b332-34c74d649ed1","Type":"ContainerStarted","Data":"a4ed9e5012f965bf1ba42214bf0e1ed496960f241f027402a208004d41956b8f"} Jul 14 15:26:34.258047 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:34.258032 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-6tgk4" Jul 14 15:26:34.272142 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:34.272090 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7f5f96f494-p85qh" podStartSLOduration=1.580902986 podStartE2EDuration="7.272073988s" totalImagesPullingTime="5.691171002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:26:34.269111292 +0000 UTC m=+335.121494456" watchObservedRunningTime="2026-07-14 15:26:34.272073988 +0000 UTC m=+335.124457154" Jul 14 15:26:34.285212 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:34.285137 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:34.293677 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:34.293621 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" podStartSLOduration=1.766651041 podStartE2EDuration="6.293600026s" totalImagesPullingTime="4.526948985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:26:34.292844919 +0000 UTC m=+335.145228083" watchObservedRunningTime="2026-07-14 15:26:34.293600026 +0000 UTC m=+335.145983192" Jul 14 15:26:34.317234 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:34.316487 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-6tgk4" podStartSLOduration=1.845216807 podStartE2EDuration="7.316466081s" totalImagesPullingTime="5.471249274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:26:34.314961549 +0000 UTC m=+335.167344715" watchObservedRunningTime="2026-07-14 15:26:34.316466081 +0000 UTC m=+335.168849248" Jul 14 15:26:34.336023 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:34.335979 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-ddsx7" podStartSLOduration=2.3116113560000002 podStartE2EDuration="6.335964017s" totalImagesPullingTime="4.024352661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:26:34.334455995 +0000 UTC m=+335.186839160" watchObservedRunningTime="2026-07-14 15:26:34.335964017 +0000 UTC m=+335.188347180" Jul 14 15:26:34.380603 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:34.380563 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:34.970158 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:34.970117 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:35.087409 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:35.087373 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:35.263671 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:35.263593 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:35.271005 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:35.270961 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:37.834741 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:37.834704 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:37.931584 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:37.931544 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:38.005324 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.005288 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:38.012609 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.012582 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67"] Jul 14 15:26:38.015988 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.015969 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67" Jul 14 15:26:38.020764 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.020744 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 14 15:26:38.020873 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.020763 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9dwkj\"" Jul 14 15:26:38.020873 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.020778 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 14 15:26:38.020873 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.020746 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 14 15:26:38.031033 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.031014 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67"] Jul 14 15:26:38.047147 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.043666 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmw97\" (UniqueName: \"kubernetes.io/projected/215eeffa-ada4-475d-b763-dedbee5cc0fc-kube-api-access-wmw97\") pod \"lws-controller-manager-d87d6d9c6-qxw67\" (UID: \"215eeffa-ada4-475d-b763-dedbee5cc0fc\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67" Jul 14 15:26:38.047147 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.043735 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/215eeffa-ada4-475d-b763-dedbee5cc0fc-cert\") pod \"lws-controller-manager-d87d6d9c6-qxw67\" (UID: \"215eeffa-ada4-475d-b763-dedbee5cc0fc\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67" Jul 14 15:26:38.047147 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.043771 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/215eeffa-ada4-475d-b763-dedbee5cc0fc-metrics-cert\") pod \"lws-controller-manager-d87d6d9c6-qxw67\" (UID: \"215eeffa-ada4-475d-b763-dedbee5cc0fc\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67" Jul 14 15:26:38.047147 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.043805 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/215eeffa-ada4-475d-b763-dedbee5cc0fc-manager-config\") pod \"lws-controller-manager-d87d6d9c6-qxw67\" (UID: \"215eeffa-ada4-475d-b763-dedbee5cc0fc\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67" Jul 14 15:26:38.145138 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.145047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmw97\" (UniqueName: \"kubernetes.io/projected/215eeffa-ada4-475d-b763-dedbee5cc0fc-kube-api-access-wmw97\") pod \"lws-controller-manager-d87d6d9c6-qxw67\" (UID: \"215eeffa-ada4-475d-b763-dedbee5cc0fc\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67" Jul 14 15:26:38.145138 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.145092 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/215eeffa-ada4-475d-b763-dedbee5cc0fc-cert\") pod \"lws-controller-manager-d87d6d9c6-qxw67\" (UID: \"215eeffa-ada4-475d-b763-dedbee5cc0fc\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67" Jul 14 15:26:38.145138 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.145125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/215eeffa-ada4-475d-b763-dedbee5cc0fc-metrics-cert\") pod \"lws-controller-manager-d87d6d9c6-qxw67\" (UID: \"215eeffa-ada4-475d-b763-dedbee5cc0fc\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67" Jul 14 15:26:38.145503 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.145147 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/215eeffa-ada4-475d-b763-dedbee5cc0fc-manager-config\") pod \"lws-controller-manager-d87d6d9c6-qxw67\" (UID: \"215eeffa-ada4-475d-b763-dedbee5cc0fc\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67" Jul 14 15:26:38.145979 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.145879 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/215eeffa-ada4-475d-b763-dedbee5cc0fc-manager-config\") pod \"lws-controller-manager-d87d6d9c6-qxw67\" (UID: \"215eeffa-ada4-475d-b763-dedbee5cc0fc\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67" Jul 14 15:26:38.147774 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.147751 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/215eeffa-ada4-475d-b763-dedbee5cc0fc-cert\") pod \"lws-controller-manager-d87d6d9c6-qxw67\" (UID: \"215eeffa-ada4-475d-b763-dedbee5cc0fc\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67" Jul 14 15:26:38.147876 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.147784 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/215eeffa-ada4-475d-b763-dedbee5cc0fc-metrics-cert\") pod \"lws-controller-manager-d87d6d9c6-qxw67\" (UID: \"215eeffa-ada4-475d-b763-dedbee5cc0fc\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67" Jul 14 15:26:38.156374 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.156341 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmw97\" (UniqueName: \"kubernetes.io/projected/215eeffa-ada4-475d-b763-dedbee5cc0fc-kube-api-access-wmw97\") pod \"lws-controller-manager-d87d6d9c6-qxw67\" (UID: \"215eeffa-ada4-475d-b763-dedbee5cc0fc\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67" Jul 14 15:26:38.325821 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.325788 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67" Jul 14 15:26:38.463084 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.463056 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67"] Jul 14 15:26:38.465037 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:26:38.465012 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215eeffa_ada4_475d_b763_dedbee5cc0fc.slice/crio-decef99bc5eb075746da97df5df4e00b3148753cb2a1842d261a248002dcef7f WatchSource:0}: Error finding container decef99bc5eb075746da97df5df4e00b3148753cb2a1842d261a248002dcef7f: Status 404 returned error can't find the container with id decef99bc5eb075746da97df5df4e00b3148753cb2a1842d261a248002dcef7f Jul 14 15:26:38.534865 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:38.534817 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:39.137865 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:39.137828 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:39.236293 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:39.236238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:26:39.280781 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:39.280749 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67" event={"ID":"215eeffa-ada4-475d-b763-dedbee5cc0fc","Type":"ContainerStarted","Data":"decef99bc5eb075746da97df5df4e00b3148753cb2a1842d261a248002dcef7f"} Jul 14 15:26:41.183414 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.183352 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n"] Jul 14 15:26:41.183849 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.183569 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" podUID="e7d58cdb-8e6f-4a39-acb2-15967443501f" containerName="discovery" containerID="cri-o://1ee837a1d9b760c0e77deaed23ba8f77775b47cd0da3d944b3e7d46a2058e6fb" gracePeriod=30 Jul 14 15:26:41.263616 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.263577 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-zxn6n container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:8080/ready\": dial tcp 10.132.0.32:8080: connect: connection refused" start-of-body= Jul 14 15:26:41.263777 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.263647 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" podUID="e7d58cdb-8e6f-4a39-acb2-15967443501f" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.32:8080/ready\": dial tcp 10.132.0.32:8080: connect: connection refused" Jul 14 15:26:41.698226 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.698202 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:41.783580 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.783552 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-csr-ca-configmap\") pod \"e7d58cdb-8e6f-4a39-acb2-15967443501f\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " Jul 14 15:26:41.783753 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.783593 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7d58cdb-8e6f-4a39-acb2-15967443501f-cacerts\") pod \"e7d58cdb-8e6f-4a39-acb2-15967443501f\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " Jul 14 15:26:41.783753 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.783617 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-token\") pod \"e7d58cdb-8e6f-4a39-acb2-15967443501f\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " Jul 14 15:26:41.783753 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.783643 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d58cdb-8e6f-4a39-acb2-15967443501f-local-certs\") pod \"e7d58cdb-8e6f-4a39-acb2-15967443501f\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " Jul 14 15:26:41.783753 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.783668 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-csr-dns-cert\") pod \"e7d58cdb-8e6f-4a39-acb2-15967443501f\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " Jul 14 15:26:41.783753 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.783700 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-kubeconfig\") pod \"e7d58cdb-8e6f-4a39-acb2-15967443501f\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " Jul 14 15:26:41.784011 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.783804 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r5lr\" (UniqueName: \"kubernetes.io/projected/e7d58cdb-8e6f-4a39-acb2-15967443501f-kube-api-access-5r5lr\") pod \"e7d58cdb-8e6f-4a39-acb2-15967443501f\" (UID: \"e7d58cdb-8e6f-4a39-acb2-15967443501f\") " Jul 14 15:26:41.784011 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.783955 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "e7d58cdb-8e6f-4a39-acb2-15967443501f" (UID: "e7d58cdb-8e6f-4a39-acb2-15967443501f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 15:26:41.784111 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.784080 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-csr-ca-configmap\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:26:41.786083 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.786050 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d58cdb-8e6f-4a39-acb2-15967443501f-cacerts" (OuterVolumeSpecName: "cacerts") pod "e7d58cdb-8e6f-4a39-acb2-15967443501f" (UID: "e7d58cdb-8e6f-4a39-acb2-15967443501f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 15:26:41.786083 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.786060 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "e7d58cdb-8e6f-4a39-acb2-15967443501f" (UID: "e7d58cdb-8e6f-4a39-acb2-15967443501f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 15:26:41.786347 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.786210 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "e7d58cdb-8e6f-4a39-acb2-15967443501f" (UID: "e7d58cdb-8e6f-4a39-acb2-15967443501f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 15:26:41.786404 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.786347 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-token" (OuterVolumeSpecName: "istio-token") pod "e7d58cdb-8e6f-4a39-acb2-15967443501f" (UID: "e7d58cdb-8e6f-4a39-acb2-15967443501f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:26:41.786404 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.786376 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d58cdb-8e6f-4a39-acb2-15967443501f-local-certs" (OuterVolumeSpecName: "local-certs") pod "e7d58cdb-8e6f-4a39-acb2-15967443501f" (UID: "e7d58cdb-8e6f-4a39-acb2-15967443501f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:26:41.786485 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.786468 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d58cdb-8e6f-4a39-acb2-15967443501f-kube-api-access-5r5lr" (OuterVolumeSpecName: "kube-api-access-5r5lr") pod "e7d58cdb-8e6f-4a39-acb2-15967443501f" (UID: "e7d58cdb-8e6f-4a39-acb2-15967443501f"). InnerVolumeSpecName "kube-api-access-5r5lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:26:41.884974 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.884939 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r5lr\" (UniqueName: \"kubernetes.io/projected/e7d58cdb-8e6f-4a39-acb2-15967443501f-kube-api-access-5r5lr\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:26:41.884974 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.884968 2579 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7d58cdb-8e6f-4a39-acb2-15967443501f-cacerts\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:26:41.884974 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.884978 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-token\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:26:41.885267 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.884986 2579 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d58cdb-8e6f-4a39-acb2-15967443501f-local-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:26:41.885267 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.884994 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-csr-dns-cert\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:26:41.885267 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:41.885002 2579 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7d58cdb-8e6f-4a39-acb2-15967443501f-istio-kubeconfig\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:26:42.298593 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:42.298558 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67" event={"ID":"215eeffa-ada4-475d-b763-dedbee5cc0fc","Type":"ContainerStarted","Data":"cddc24f834b34684f943a7ba61b13a5acdd5f9c191aa5e5bba0b8eb9327f5ae5"} Jul 14 15:26:42.299023 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:42.298707 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67" Jul 14 15:26:42.299842 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:42.299814 2579 generic.go:356] "Generic (PLEG): container finished" podID="e7d58cdb-8e6f-4a39-acb2-15967443501f" containerID="1ee837a1d9b760c0e77deaed23ba8f77775b47cd0da3d944b3e7d46a2058e6fb" exitCode=0 Jul 14 15:26:42.299982 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:42.299852 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" Jul 14 15:26:42.299982 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:42.299867 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" event={"ID":"e7d58cdb-8e6f-4a39-acb2-15967443501f","Type":"ContainerDied","Data":"1ee837a1d9b760c0e77deaed23ba8f77775b47cd0da3d944b3e7d46a2058e6fb"} Jul 14 15:26:42.299982 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:42.299911 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n" event={"ID":"e7d58cdb-8e6f-4a39-acb2-15967443501f","Type":"ContainerDied","Data":"ca83b6347ba0b7153b8540140471e91791303db5867d50655f70fccbc535d73e"} Jul 14 15:26:42.299982 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:42.299940 2579 scope.go:117] "RemoveContainer" containerID="1ee837a1d9b760c0e77deaed23ba8f77775b47cd0da3d944b3e7d46a2058e6fb" Jul 14 15:26:42.308810 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:42.308796 2579 scope.go:117] "RemoveContainer" containerID="1ee837a1d9b760c0e77deaed23ba8f77775b47cd0da3d944b3e7d46a2058e6fb" Jul 14 15:26:42.309043 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:26:42.309022 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee837a1d9b760c0e77deaed23ba8f77775b47cd0da3d944b3e7d46a2058e6fb\": container with ID starting with 1ee837a1d9b760c0e77deaed23ba8f77775b47cd0da3d944b3e7d46a2058e6fb not found: ID does not exist" containerID="1ee837a1d9b760c0e77deaed23ba8f77775b47cd0da3d944b3e7d46a2058e6fb" Jul 14 15:26:42.309084 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:42.309054 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee837a1d9b760c0e77deaed23ba8f77775b47cd0da3d944b3e7d46a2058e6fb"} err="failed to get container status \"1ee837a1d9b760c0e77deaed23ba8f77775b47cd0da3d944b3e7d46a2058e6fb\": rpc error: code = NotFound desc = could not find container \"1ee837a1d9b760c0e77deaed23ba8f77775b47cd0da3d944b3e7d46a2058e6fb\": container with ID starting with 1ee837a1d9b760c0e77deaed23ba8f77775b47cd0da3d944b3e7d46a2058e6fb not found: ID does not exist" Jul 14 15:26:42.320043 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:42.320004 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67" podStartSLOduration=2.177318519 podStartE2EDuration="5.319990431s" totalImagesPullingTime="3.142671912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:26:42.318517422 +0000 UTC m=+343.170900588" watchObservedRunningTime="2026-07-14 15:26:42.319990431 +0000 UTC m=+343.172373595" Jul 14 15:26:42.335973 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:42.335949 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n"] Jul 14 15:26:42.343250 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:42.343230 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zxn6n"] Jul 14 15:26:43.696355 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:43.696320 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7d58cdb-8e6f-4a39-acb2-15967443501f" path="/var/lib/kubelet/pods/e7d58cdb-8e6f-4a39-acb2-15967443501f/volumes" Jul 14 15:26:45.265789 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:45.265756 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-6tgk4" Jul 14 15:26:45.268519 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:45.268498 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-ddsx7" Jul 14 15:26:53.308316 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:26:53.308242 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-qxw67" Jul 14 15:27:15.381218 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:15.381164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:27:28.558083 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:28.558025 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:27:28.660339 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:28.660304 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:27:28.720527 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:28.720492 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:27:29.454054 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:29.454001 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:27:29.680102 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:29.680058 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:27:44.495590 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.495550 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr"] Jul 14 15:27:44.496055 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.496017 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d58cdb-8e6f-4a39-acb2-15967443501f" containerName="discovery" Jul 14 15:27:44.496055 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.496036 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d58cdb-8e6f-4a39-acb2-15967443501f" containerName="discovery" Jul 14 15:27:44.496206 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.496132 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7d58cdb-8e6f-4a39-acb2-15967443501f" containerName="discovery" Jul 14 15:27:44.499428 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.499403 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr" Jul 14 15:27:44.502422 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.502401 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 15:27:44.502634 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.502622 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 15:27:44.504207 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.504185 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-t6r2p\"" Jul 14 15:27:44.506973 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.506952 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr"] Jul 14 15:27:44.618744 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.618711 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da3c5761-0462-4689-b1ea-968bb6c0d5de-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr\" (UID: \"da3c5761-0462-4689-b1ea-968bb6c0d5de\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr" Jul 14 15:27:44.618934 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.618784 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7vlq\" (UniqueName: \"kubernetes.io/projected/da3c5761-0462-4689-b1ea-968bb6c0d5de-kube-api-access-v7vlq\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr\" (UID: \"da3c5761-0462-4689-b1ea-968bb6c0d5de\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr" Jul 14 15:27:44.618934 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.618831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da3c5761-0462-4689-b1ea-968bb6c0d5de-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr\" (UID: \"da3c5761-0462-4689-b1ea-968bb6c0d5de\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr" Jul 14 15:27:44.719610 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.719575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7vlq\" (UniqueName: \"kubernetes.io/projected/da3c5761-0462-4689-b1ea-968bb6c0d5de-kube-api-access-v7vlq\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr\" (UID: \"da3c5761-0462-4689-b1ea-968bb6c0d5de\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr" Jul 14 15:27:44.719797 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.719636 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da3c5761-0462-4689-b1ea-968bb6c0d5de-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr\" (UID: \"da3c5761-0462-4689-b1ea-968bb6c0d5de\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr" Jul 14 15:27:44.719797 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.719685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da3c5761-0462-4689-b1ea-968bb6c0d5de-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr\" (UID: \"da3c5761-0462-4689-b1ea-968bb6c0d5de\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr" Jul 14 15:27:44.720001 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.719978 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da3c5761-0462-4689-b1ea-968bb6c0d5de-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr\" (UID: \"da3c5761-0462-4689-b1ea-968bb6c0d5de\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr" Jul 14 15:27:44.720069 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.720044 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da3c5761-0462-4689-b1ea-968bb6c0d5de-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr\" (UID: \"da3c5761-0462-4689-b1ea-968bb6c0d5de\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr" Jul 14 15:27:44.729836 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.729804 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7vlq\" (UniqueName: \"kubernetes.io/projected/da3c5761-0462-4689-b1ea-968bb6c0d5de-kube-api-access-v7vlq\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr\" (UID: \"da3c5761-0462-4689-b1ea-968bb6c0d5de\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr" Jul 14 15:27:44.810018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.809937 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr" Jul 14 15:27:44.899586 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.899555 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c"] Jul 14 15:27:44.904650 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.904625 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c" Jul 14 15:27:44.910914 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.910861 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c"] Jul 14 15:27:44.942259 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:44.942236 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr"] Jul 14 15:27:44.944044 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:27:44.944019 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3c5761_0462_4689_b1ea_968bb6c0d5de.slice/crio-de6cc6896c6dd1bbb8747f87c9660cdff382d2342cb32b351e51b689089ae9cd WatchSource:0}: Error finding container de6cc6896c6dd1bbb8747f87c9660cdff382d2342cb32b351e51b689089ae9cd: Status 404 returned error can't find the container with id de6cc6896c6dd1bbb8747f87c9660cdff382d2342cb32b351e51b689089ae9cd Jul 14 15:27:45.023082 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.023043 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc600c37-b36e-4d51-9160-f91d1e881813-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c\" (UID: \"bc600c37-b36e-4d51-9160-f91d1e881813\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c" Jul 14 15:27:45.023275 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.023085 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njk7r\" (UniqueName: \"kubernetes.io/projected/bc600c37-b36e-4d51-9160-f91d1e881813-kube-api-access-njk7r\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c\" (UID: \"bc600c37-b36e-4d51-9160-f91d1e881813\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c" Jul 14 15:27:45.023275 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.023135 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc600c37-b36e-4d51-9160-f91d1e881813-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c\" (UID: \"bc600c37-b36e-4d51-9160-f91d1e881813\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c" Jul 14 15:27:45.124273 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.124241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc600c37-b36e-4d51-9160-f91d1e881813-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c\" (UID: \"bc600c37-b36e-4d51-9160-f91d1e881813\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c" Jul 14 15:27:45.124437 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.124285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njk7r\" (UniqueName: \"kubernetes.io/projected/bc600c37-b36e-4d51-9160-f91d1e881813-kube-api-access-njk7r\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c\" (UID: \"bc600c37-b36e-4d51-9160-f91d1e881813\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c" Jul 14 15:27:45.124437 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.124315 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc600c37-b36e-4d51-9160-f91d1e881813-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c\" (UID: \"bc600c37-b36e-4d51-9160-f91d1e881813\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c" Jul 14 15:27:45.124629 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.124606 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc600c37-b36e-4d51-9160-f91d1e881813-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c\" (UID: \"bc600c37-b36e-4d51-9160-f91d1e881813\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c" Jul 14 15:27:45.124664 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.124652 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc600c37-b36e-4d51-9160-f91d1e881813-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c\" (UID: \"bc600c37-b36e-4d51-9160-f91d1e881813\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c" Jul 14 15:27:45.135394 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.135366 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njk7r\" (UniqueName: \"kubernetes.io/projected/bc600c37-b36e-4d51-9160-f91d1e881813-kube-api-access-njk7r\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c\" (UID: \"bc600c37-b36e-4d51-9160-f91d1e881813\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c" Jul 14 15:27:45.218271 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.218234 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c" Jul 14 15:27:45.295188 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.295138 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv"] Jul 14 15:27:45.299662 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.299640 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv" Jul 14 15:27:45.307075 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.306777 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv"] Jul 14 15:27:45.354472 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.354447 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c"] Jul 14 15:27:45.356341 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:27:45.356313 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc600c37_b36e_4d51_9160_f91d1e881813.slice/crio-f5cb417c18b99e8ae1388cd21ab68462a357e9b8220ba280b5d31ccdc45f4f3e WatchSource:0}: Error finding container f5cb417c18b99e8ae1388cd21ab68462a357e9b8220ba280b5d31ccdc45f4f3e: Status 404 returned error can't find the container with id f5cb417c18b99e8ae1388cd21ab68462a357e9b8220ba280b5d31ccdc45f4f3e Jul 14 15:27:45.427116 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.427088 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9xjf\" (UniqueName: \"kubernetes.io/projected/83061462-6051-466b-9e61-90164f652346-kube-api-access-x9xjf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv\" (UID: \"83061462-6051-466b-9e61-90164f652346\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv" Jul 14 15:27:45.427245 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.427126 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83061462-6051-466b-9e61-90164f652346-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv\" (UID: \"83061462-6051-466b-9e61-90164f652346\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv" Jul 14 15:27:45.427245 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.427220 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83061462-6051-466b-9e61-90164f652346-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv\" (UID: \"83061462-6051-466b-9e61-90164f652346\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv" Jul 14 15:27:45.528041 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.528005 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9xjf\" (UniqueName: \"kubernetes.io/projected/83061462-6051-466b-9e61-90164f652346-kube-api-access-x9xjf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv\" (UID: \"83061462-6051-466b-9e61-90164f652346\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv" Jul 14 15:27:45.528522 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.528048 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83061462-6051-466b-9e61-90164f652346-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv\" (UID: \"83061462-6051-466b-9e61-90164f652346\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv" Jul 14 15:27:45.528522 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.528091 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83061462-6051-466b-9e61-90164f652346-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv\" (UID: \"83061462-6051-466b-9e61-90164f652346\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv" Jul 14 15:27:45.528522 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.528505 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83061462-6051-466b-9e61-90164f652346-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv\" (UID: \"83061462-6051-466b-9e61-90164f652346\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv" Jul 14 15:27:45.528662 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.528559 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83061462-6051-466b-9e61-90164f652346-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv\" (UID: \"83061462-6051-466b-9e61-90164f652346\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv" Jul 14 15:27:45.537437 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.537411 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9xjf\" (UniqueName: \"kubernetes.io/projected/83061462-6051-466b-9e61-90164f652346-kube-api-access-x9xjf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv\" (UID: \"83061462-6051-466b-9e61-90164f652346\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv" Jul 14 15:27:45.599080 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.599045 2579 generic.go:356] "Generic (PLEG): container finished" podID="da3c5761-0462-4689-b1ea-968bb6c0d5de" containerID="0be5a347d035cb919c3f40ee320c2f264e2ad4e3ff6b942fdfc2640f5c721283" exitCode=0 Jul 14 15:27:45.599278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.599136 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr" event={"ID":"da3c5761-0462-4689-b1ea-968bb6c0d5de","Type":"ContainerDied","Data":"0be5a347d035cb919c3f40ee320c2f264e2ad4e3ff6b942fdfc2640f5c721283"} Jul 14 15:27:45.599278 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.599211 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr" event={"ID":"da3c5761-0462-4689-b1ea-968bb6c0d5de","Type":"ContainerStarted","Data":"de6cc6896c6dd1bbb8747f87c9660cdff382d2342cb32b351e51b689089ae9cd"} Jul 14 15:27:45.600737 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.600714 2579 generic.go:356] "Generic (PLEG): container finished" podID="bc600c37-b36e-4d51-9160-f91d1e881813" containerID="aeddd65aa602726a6505cb708748cf1f20fda8874c4b1191826db8276daa5fc9" exitCode=0 Jul 14 15:27:45.600842 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.600797 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c" event={"ID":"bc600c37-b36e-4d51-9160-f91d1e881813","Type":"ContainerDied","Data":"aeddd65aa602726a6505cb708748cf1f20fda8874c4b1191826db8276daa5fc9"} Jul 14 15:27:45.600895 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.600845 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c" event={"ID":"bc600c37-b36e-4d51-9160-f91d1e881813","Type":"ContainerStarted","Data":"f5cb417c18b99e8ae1388cd21ab68462a357e9b8220ba280b5d31ccdc45f4f3e"} Jul 14 15:27:45.612767 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.612749 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv" Jul 14 15:27:45.699893 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.699866 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5"] Jul 14 15:27:45.704468 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.704446 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5" Jul 14 15:27:45.713775 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.712946 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5"] Jul 14 15:27:45.746050 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.746024 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv"] Jul 14 15:27:45.747628 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:27:45.747606 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83061462_6051_466b_9e61_90164f652346.slice/crio-015803185f16b3875c563d917c6d69211bdf71b25e0abf28deec2d939a16f5e0 WatchSource:0}: Error finding container 015803185f16b3875c563d917c6d69211bdf71b25e0abf28deec2d939a16f5e0: Status 404 returned error can't find the container with id 015803185f16b3875c563d917c6d69211bdf71b25e0abf28deec2d939a16f5e0 Jul 14 15:27:45.829707 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.829677 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb58ab6d-8105-41a1-9c2f-6d74e1010daa-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5\" (UID: \"cb58ab6d-8105-41a1-9c2f-6d74e1010daa\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5" Jul 14 15:27:45.829848 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.829741 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb58ab6d-8105-41a1-9c2f-6d74e1010daa-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5\" (UID: \"cb58ab6d-8105-41a1-9c2f-6d74e1010daa\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5" Jul 14 15:27:45.829910 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.829883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhfps\" (UniqueName: \"kubernetes.io/projected/cb58ab6d-8105-41a1-9c2f-6d74e1010daa-kube-api-access-zhfps\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5\" (UID: \"cb58ab6d-8105-41a1-9c2f-6d74e1010daa\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5" Jul 14 15:27:45.930421 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.930343 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb58ab6d-8105-41a1-9c2f-6d74e1010daa-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5\" (UID: \"cb58ab6d-8105-41a1-9c2f-6d74e1010daa\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5" Jul 14 15:27:45.930421 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.930413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhfps\" (UniqueName: \"kubernetes.io/projected/cb58ab6d-8105-41a1-9c2f-6d74e1010daa-kube-api-access-zhfps\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5\" (UID: \"cb58ab6d-8105-41a1-9c2f-6d74e1010daa\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5" Jul 14 15:27:45.930747 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.930439 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb58ab6d-8105-41a1-9c2f-6d74e1010daa-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5\" (UID: \"cb58ab6d-8105-41a1-9c2f-6d74e1010daa\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5" Jul 14 15:27:45.930747 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.930707 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb58ab6d-8105-41a1-9c2f-6d74e1010daa-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5\" (UID: \"cb58ab6d-8105-41a1-9c2f-6d74e1010daa\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5" Jul 14 15:27:45.930810 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.930760 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb58ab6d-8105-41a1-9c2f-6d74e1010daa-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5\" (UID: \"cb58ab6d-8105-41a1-9c2f-6d74e1010daa\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5" Jul 14 15:27:45.940128 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:45.940101 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhfps\" (UniqueName: \"kubernetes.io/projected/cb58ab6d-8105-41a1-9c2f-6d74e1010daa-kube-api-access-zhfps\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5\" (UID: \"cb58ab6d-8105-41a1-9c2f-6d74e1010daa\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5" Jul 14 15:27:46.017048 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:46.017021 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5" Jul 14 15:27:46.147916 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:46.147889 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5"] Jul 14 15:27:46.149650 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:27:46.149618 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb58ab6d_8105_41a1_9c2f_6d74e1010daa.slice/crio-73c76da46748e44ca6bfdea97dd33e14405651073d87199dba55ee95cd135074 WatchSource:0}: Error finding container 73c76da46748e44ca6bfdea97dd33e14405651073d87199dba55ee95cd135074: Status 404 returned error can't find the container with id 73c76da46748e44ca6bfdea97dd33e14405651073d87199dba55ee95cd135074 Jul 14 15:27:46.608896 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:46.608848 2579 generic.go:356] "Generic (PLEG): container finished" podID="cb58ab6d-8105-41a1-9c2f-6d74e1010daa" containerID="11123e0a3a1c78368163328a7d17187bbd4ba6036177232d85bf46082dc20f1a" exitCode=0 Jul 14 15:27:46.609424 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:46.608920 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5" event={"ID":"cb58ab6d-8105-41a1-9c2f-6d74e1010daa","Type":"ContainerDied","Data":"11123e0a3a1c78368163328a7d17187bbd4ba6036177232d85bf46082dc20f1a"} Jul 14 15:27:46.609424 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:46.609023 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5" event={"ID":"cb58ab6d-8105-41a1-9c2f-6d74e1010daa","Type":"ContainerStarted","Data":"73c76da46748e44ca6bfdea97dd33e14405651073d87199dba55ee95cd135074"} Jul 14 15:27:46.610801 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:46.610774 2579 generic.go:356] "Generic (PLEG): container finished" podID="83061462-6051-466b-9e61-90164f652346" containerID="34088de7015d0427c2482e151bd7f48e705d1d804ad51f97fd9ed11d84036788" exitCode=0 Jul 14 15:27:46.610895 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:46.610873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv" event={"ID":"83061462-6051-466b-9e61-90164f652346","Type":"ContainerDied","Data":"34088de7015d0427c2482e151bd7f48e705d1d804ad51f97fd9ed11d84036788"} Jul 14 15:27:46.610956 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:46.610908 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv" event={"ID":"83061462-6051-466b-9e61-90164f652346","Type":"ContainerStarted","Data":"015803185f16b3875c563d917c6d69211bdf71b25e0abf28deec2d939a16f5e0"} Jul 14 15:27:47.617343 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:47.617308 2579 generic.go:356] "Generic (PLEG): container finished" podID="da3c5761-0462-4689-b1ea-968bb6c0d5de" containerID="cd1a1f48a7235b9a67b2783e9cdd80f1e74008dd163b1a82d49277eae1a537e8" exitCode=0 Jul 14 15:27:47.617748 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:47.617402 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr" event={"ID":"da3c5761-0462-4689-b1ea-968bb6c0d5de","Type":"ContainerDied","Data":"cd1a1f48a7235b9a67b2783e9cdd80f1e74008dd163b1a82d49277eae1a537e8"} Jul 14 15:27:47.619297 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:47.619274 2579 generic.go:356] "Generic (PLEG): container finished" podID="bc600c37-b36e-4d51-9160-f91d1e881813" containerID="221542391f4fbbb6b24efae883eec6b53e607a4068c803f602b1defab9df60ee" exitCode=0 Jul 14 15:27:47.619380 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:47.619358 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c" event={"ID":"bc600c37-b36e-4d51-9160-f91d1e881813","Type":"ContainerDied","Data":"221542391f4fbbb6b24efae883eec6b53e607a4068c803f602b1defab9df60ee"} Jul 14 15:27:47.621132 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:47.621110 2579 generic.go:356] "Generic (PLEG): container finished" podID="cb58ab6d-8105-41a1-9c2f-6d74e1010daa" containerID="13b931e50efdae499a72aaa3995380afcd54513aead174eeba00e1227499b618" exitCode=0 Jul 14 15:27:47.621240 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:47.621189 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5" event={"ID":"cb58ab6d-8105-41a1-9c2f-6d74e1010daa","Type":"ContainerDied","Data":"13b931e50efdae499a72aaa3995380afcd54513aead174eeba00e1227499b618"} Jul 14 15:27:48.628307 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:48.628270 2579 generic.go:356] "Generic (PLEG): container finished" podID="da3c5761-0462-4689-b1ea-968bb6c0d5de" containerID="b54bb78580bf8116c78f7c101b9ee37d60595ba032ab26248171bcf7854f92a3" exitCode=0 Jul 14 15:27:48.628702 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:48.628356 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr" event={"ID":"da3c5761-0462-4689-b1ea-968bb6c0d5de","Type":"ContainerDied","Data":"b54bb78580bf8116c78f7c101b9ee37d60595ba032ab26248171bcf7854f92a3"} Jul 14 15:27:48.630205 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:48.630183 2579 generic.go:356] "Generic (PLEG): container finished" podID="bc600c37-b36e-4d51-9160-f91d1e881813" containerID="71d10e4f26d1d3a97bd01dcf8aada11a72e1b0c80d0211eada856be40428c44e" exitCode=0 Jul 14 15:27:48.630205 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:48.630194 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c" event={"ID":"bc600c37-b36e-4d51-9160-f91d1e881813","Type":"ContainerDied","Data":"71d10e4f26d1d3a97bd01dcf8aada11a72e1b0c80d0211eada856be40428c44e"} Jul 14 15:27:48.632084 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:48.632067 2579 generic.go:356] "Generic (PLEG): container finished" podID="cb58ab6d-8105-41a1-9c2f-6d74e1010daa" containerID="fa24a45d3e2d1038bf8ffc98469fdcbbc74abefff532f229ef1c6ba0ed3afa35" exitCode=0 Jul 14 15:27:48.632127 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:48.632117 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5" event={"ID":"cb58ab6d-8105-41a1-9c2f-6d74e1010daa","Type":"ContainerDied","Data":"fa24a45d3e2d1038bf8ffc98469fdcbbc74abefff532f229ef1c6ba0ed3afa35"} Jul 14 15:27:49.638086 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.637995 2579 generic.go:356] "Generic (PLEG): container finished" podID="83061462-6051-466b-9e61-90164f652346" containerID="39f25c341832b03a5ffd73d54b5aed81cc113b1a71829532febc3c81dfe76aaf" exitCode=0 Jul 14 15:27:49.638497 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.638082 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv" event={"ID":"83061462-6051-466b-9e61-90164f652346","Type":"ContainerDied","Data":"39f25c341832b03a5ffd73d54b5aed81cc113b1a71829532febc3c81dfe76aaf"} Jul 14 15:27:49.855408 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.855384 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5" Jul 14 15:27:49.858845 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.858816 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr" Jul 14 15:27:49.861762 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.861739 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c" Jul 14 15:27:49.964652 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.964576 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da3c5761-0462-4689-b1ea-968bb6c0d5de-util\") pod \"da3c5761-0462-4689-b1ea-968bb6c0d5de\" (UID: \"da3c5761-0462-4689-b1ea-968bb6c0d5de\") " Jul 14 15:27:49.964652 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.964619 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb58ab6d-8105-41a1-9c2f-6d74e1010daa-bundle\") pod \"cb58ab6d-8105-41a1-9c2f-6d74e1010daa\" (UID: \"cb58ab6d-8105-41a1-9c2f-6d74e1010daa\") " Jul 14 15:27:49.964652 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.964638 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb58ab6d-8105-41a1-9c2f-6d74e1010daa-util\") pod \"cb58ab6d-8105-41a1-9c2f-6d74e1010daa\" (UID: \"cb58ab6d-8105-41a1-9c2f-6d74e1010daa\") " Jul 14 15:27:49.964870 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.964662 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da3c5761-0462-4689-b1ea-968bb6c0d5de-bundle\") pod \"da3c5761-0462-4689-b1ea-968bb6c0d5de\" (UID: \"da3c5761-0462-4689-b1ea-968bb6c0d5de\") " Jul 14 15:27:49.964870 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.964693 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njk7r\" (UniqueName: \"kubernetes.io/projected/bc600c37-b36e-4d51-9160-f91d1e881813-kube-api-access-njk7r\") pod \"bc600c37-b36e-4d51-9160-f91d1e881813\" (UID: \"bc600c37-b36e-4d51-9160-f91d1e881813\") " Jul 14 15:27:49.964870 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.964745 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhfps\" (UniqueName: \"kubernetes.io/projected/cb58ab6d-8105-41a1-9c2f-6d74e1010daa-kube-api-access-zhfps\") pod \"cb58ab6d-8105-41a1-9c2f-6d74e1010daa\" (UID: \"cb58ab6d-8105-41a1-9c2f-6d74e1010daa\") " Jul 14 15:27:49.964870 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.964779 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7vlq\" (UniqueName: \"kubernetes.io/projected/da3c5761-0462-4689-b1ea-968bb6c0d5de-kube-api-access-v7vlq\") pod \"da3c5761-0462-4689-b1ea-968bb6c0d5de\" (UID: \"da3c5761-0462-4689-b1ea-968bb6c0d5de\") " Jul 14 15:27:49.964870 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.964822 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc600c37-b36e-4d51-9160-f91d1e881813-util\") pod \"bc600c37-b36e-4d51-9160-f91d1e881813\" (UID: \"bc600c37-b36e-4d51-9160-f91d1e881813\") " Jul 14 15:27:49.965052 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.964871 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc600c37-b36e-4d51-9160-f91d1e881813-bundle\") pod \"bc600c37-b36e-4d51-9160-f91d1e881813\" (UID: \"bc600c37-b36e-4d51-9160-f91d1e881813\") " Jul 14 15:27:49.965195 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.965120 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3c5761-0462-4689-b1ea-968bb6c0d5de-bundle" (OuterVolumeSpecName: "bundle") pod "da3c5761-0462-4689-b1ea-968bb6c0d5de" (UID: "da3c5761-0462-4689-b1ea-968bb6c0d5de"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:27:49.965567 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.965542 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb58ab6d-8105-41a1-9c2f-6d74e1010daa-bundle" (OuterVolumeSpecName: "bundle") pod "cb58ab6d-8105-41a1-9c2f-6d74e1010daa" (UID: "cb58ab6d-8105-41a1-9c2f-6d74e1010daa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:27:49.965697 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.965674 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc600c37-b36e-4d51-9160-f91d1e881813-bundle" (OuterVolumeSpecName: "bundle") pod "bc600c37-b36e-4d51-9160-f91d1e881813" (UID: "bc600c37-b36e-4d51-9160-f91d1e881813"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:27:49.967451 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.967416 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb58ab6d-8105-41a1-9c2f-6d74e1010daa-kube-api-access-zhfps" (OuterVolumeSpecName: "kube-api-access-zhfps") pod "cb58ab6d-8105-41a1-9c2f-6d74e1010daa" (UID: "cb58ab6d-8105-41a1-9c2f-6d74e1010daa"). InnerVolumeSpecName "kube-api-access-zhfps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:27:49.967568 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.967548 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3c5761-0462-4689-b1ea-968bb6c0d5de-kube-api-access-v7vlq" (OuterVolumeSpecName: "kube-api-access-v7vlq") pod "da3c5761-0462-4689-b1ea-968bb6c0d5de" (UID: "da3c5761-0462-4689-b1ea-968bb6c0d5de"). InnerVolumeSpecName "kube-api-access-v7vlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:27:49.967726 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.967706 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc600c37-b36e-4d51-9160-f91d1e881813-kube-api-access-njk7r" (OuterVolumeSpecName: "kube-api-access-njk7r") pod "bc600c37-b36e-4d51-9160-f91d1e881813" (UID: "bc600c37-b36e-4d51-9160-f91d1e881813"). InnerVolumeSpecName "kube-api-access-njk7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:27:49.970202 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.970165 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb58ab6d-8105-41a1-9c2f-6d74e1010daa-util" (OuterVolumeSpecName: "util") pod "cb58ab6d-8105-41a1-9c2f-6d74e1010daa" (UID: "cb58ab6d-8105-41a1-9c2f-6d74e1010daa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:27:49.971971 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.971952 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3c5761-0462-4689-b1ea-968bb6c0d5de-util" (OuterVolumeSpecName: "util") pod "da3c5761-0462-4689-b1ea-968bb6c0d5de" (UID: "da3c5761-0462-4689-b1ea-968bb6c0d5de"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:27:49.972149 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:49.972129 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc600c37-b36e-4d51-9160-f91d1e881813-util" (OuterVolumeSpecName: "util") pod "bc600c37-b36e-4d51-9160-f91d1e881813" (UID: "bc600c37-b36e-4d51-9160-f91d1e881813"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:27:50.066521 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:50.066486 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc600c37-b36e-4d51-9160-f91d1e881813-bundle\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:27:50.066521 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:50.066513 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da3c5761-0462-4689-b1ea-968bb6c0d5de-util\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:27:50.066694 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:50.066521 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb58ab6d-8105-41a1-9c2f-6d74e1010daa-bundle\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:27:50.066694 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:50.066541 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb58ab6d-8105-41a1-9c2f-6d74e1010daa-util\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:27:50.066694 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:50.066549 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da3c5761-0462-4689-b1ea-968bb6c0d5de-bundle\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:27:50.066694 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:50.066557 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njk7r\" (UniqueName: \"kubernetes.io/projected/bc600c37-b36e-4d51-9160-f91d1e881813-kube-api-access-njk7r\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:27:50.066694 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:50.066566 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhfps\" (UniqueName: \"kubernetes.io/projected/cb58ab6d-8105-41a1-9c2f-6d74e1010daa-kube-api-access-zhfps\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:27:50.066694 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:50.066575 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7vlq\" (UniqueName: \"kubernetes.io/projected/da3c5761-0462-4689-b1ea-968bb6c0d5de-kube-api-access-v7vlq\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:27:50.066694 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:50.066584 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc600c37-b36e-4d51-9160-f91d1e881813-util\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:27:50.644738 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:50.644658 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr" Jul 14 15:27:50.645145 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:50.644656 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr" event={"ID":"da3c5761-0462-4689-b1ea-968bb6c0d5de","Type":"ContainerDied","Data":"de6cc6896c6dd1bbb8747f87c9660cdff382d2342cb32b351e51b689089ae9cd"} Jul 14 15:27:50.645145 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:50.644787 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de6cc6896c6dd1bbb8747f87c9660cdff382d2342cb32b351e51b689089ae9cd" Jul 14 15:27:50.646644 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:50.646626 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c" Jul 14 15:27:50.646767 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:50.646664 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c" event={"ID":"bc600c37-b36e-4d51-9160-f91d1e881813","Type":"ContainerDied","Data":"f5cb417c18b99e8ae1388cd21ab68462a357e9b8220ba280b5d31ccdc45f4f3e"} Jul 14 15:27:50.646767 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:50.646703 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5cb417c18b99e8ae1388cd21ab68462a357e9b8220ba280b5d31ccdc45f4f3e" Jul 14 15:27:50.648568 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:50.648548 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5" Jul 14 15:27:50.648670 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:50.648566 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5" event={"ID":"cb58ab6d-8105-41a1-9c2f-6d74e1010daa","Type":"ContainerDied","Data":"73c76da46748e44ca6bfdea97dd33e14405651073d87199dba55ee95cd135074"} Jul 14 15:27:50.648670 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:50.648601 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73c76da46748e44ca6bfdea97dd33e14405651073d87199dba55ee95cd135074" Jul 14 15:27:50.650735 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:50.650712 2579 generic.go:356] "Generic (PLEG): container finished" podID="83061462-6051-466b-9e61-90164f652346" containerID="2b16504d343a43d3c4e074db68ab29a6593093f75829421019efda0210c1a99e" exitCode=0 Jul 14 15:27:50.650854 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:50.650751 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv" event={"ID":"83061462-6051-466b-9e61-90164f652346","Type":"ContainerDied","Data":"2b16504d343a43d3c4e074db68ab29a6593093f75829421019efda0210c1a99e"} Jul 14 15:27:51.788378 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:51.788355 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv" Jul 14 15:27:51.883789 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:51.883753 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83061462-6051-466b-9e61-90164f652346-util\") pod \"83061462-6051-466b-9e61-90164f652346\" (UID: \"83061462-6051-466b-9e61-90164f652346\") " Jul 14 15:27:51.883965 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:51.883862 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83061462-6051-466b-9e61-90164f652346-bundle\") pod \"83061462-6051-466b-9e61-90164f652346\" (UID: \"83061462-6051-466b-9e61-90164f652346\") " Jul 14 15:27:51.883965 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:51.883890 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9xjf\" (UniqueName: \"kubernetes.io/projected/83061462-6051-466b-9e61-90164f652346-kube-api-access-x9xjf\") pod \"83061462-6051-466b-9e61-90164f652346\" (UID: \"83061462-6051-466b-9e61-90164f652346\") " Jul 14 15:27:51.884411 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:51.884375 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83061462-6051-466b-9e61-90164f652346-bundle" (OuterVolumeSpecName: "bundle") pod "83061462-6051-466b-9e61-90164f652346" (UID: "83061462-6051-466b-9e61-90164f652346"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:27:51.886091 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:51.886061 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83061462-6051-466b-9e61-90164f652346-kube-api-access-x9xjf" (OuterVolumeSpecName: "kube-api-access-x9xjf") pod "83061462-6051-466b-9e61-90164f652346" (UID: "83061462-6051-466b-9e61-90164f652346"). InnerVolumeSpecName "kube-api-access-x9xjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:27:51.888501 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:51.888465 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83061462-6051-466b-9e61-90164f652346-util" (OuterVolumeSpecName: "util") pod "83061462-6051-466b-9e61-90164f652346" (UID: "83061462-6051-466b-9e61-90164f652346"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:27:51.985220 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:51.985119 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83061462-6051-466b-9e61-90164f652346-util\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:27:51.985220 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:51.985151 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83061462-6051-466b-9e61-90164f652346-bundle\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:27:51.985220 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:51.985167 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9xjf\" (UniqueName: \"kubernetes.io/projected/83061462-6051-466b-9e61-90164f652346-kube-api-access-x9xjf\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:27:52.662125 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:52.662088 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv" event={"ID":"83061462-6051-466b-9e61-90164f652346","Type":"ContainerDied","Data":"015803185f16b3875c563d917c6d69211bdf71b25e0abf28deec2d939a16f5e0"} Jul 14 15:27:52.662125 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:52.662115 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv" Jul 14 15:27:52.662340 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:27:52.662164 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="015803185f16b3875c563d917c6d69211bdf71b25e0abf28deec2d939a16f5e0" Jul 14 15:28:03.549935 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:03.549892 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:03.607495 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:03.607453 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:07.380336 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:07.380292 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:13.791279 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:13.791239 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:16.967481 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:16.967432 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:17.235404 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:17.235315 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:17.778456 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:17.778425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:18.822530 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:18.822492 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:21.214412 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:21.214332 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:23.252770 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:23.252728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:23.575709 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:23.575672 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:24.586997 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:24.586966 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:24.691927 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:24.691888 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:28.574472 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:28.574418 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:28.675599 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:28.675558 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:36.782936 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:36.782895 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:36.878067 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:36.878029 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:44.807889 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:44.807847 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:46.373971 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:46.373932 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:46.475463 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:46.475419 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:50.014644 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:50.014610 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:50.080148 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:50.080110 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:55.981798 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:55.981756 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:56.769347 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:56.769313 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:56.873227 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:56.873191 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:57.030957 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:57.030868 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:28:57.093109 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:28:57.093052 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:01.570708 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:01.570669 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:01.671462 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:01.671423 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:04.181105 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:04.181062 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:04.281321 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:04.281293 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:07.704137 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:07.704100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:07.806137 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:07.806097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:07.865898 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:07.865865 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:07.977995 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:07.977903 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:08.078841 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:08.078805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:16.197931 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:16.197895 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:16.328651 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:16.328615 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:20.985084 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:20.985030 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:21.085526 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:21.085485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:21.215657 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:21.215615 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:21.480225 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:21.480192 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:22.926481 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:22.926447 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:23.024988 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.024953 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:23.082030 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.081993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:23.262268 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262162 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:29:23.262589 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262571 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc600c37-b36e-4d51-9160-f91d1e881813" containerName="extract" Jul 14 15:29:23.262672 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262592 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc600c37-b36e-4d51-9160-f91d1e881813" containerName="extract" Jul 14 15:29:23.262672 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262605 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb58ab6d-8105-41a1-9c2f-6d74e1010daa" containerName="extract" Jul 14 15:29:23.262672 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262613 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb58ab6d-8105-41a1-9c2f-6d74e1010daa" containerName="extract" Jul 14 15:29:23.262672 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262628 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb58ab6d-8105-41a1-9c2f-6d74e1010daa" containerName="pull" Jul 14 15:29:23.262672 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262636 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb58ab6d-8105-41a1-9c2f-6d74e1010daa" containerName="pull" Jul 14 15:29:23.262672 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262649 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb58ab6d-8105-41a1-9c2f-6d74e1010daa" containerName="util" Jul 14 15:29:23.262672 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262657 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb58ab6d-8105-41a1-9c2f-6d74e1010daa" containerName="util" Jul 14 15:29:23.263018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262677 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3c5761-0462-4689-b1ea-968bb6c0d5de" containerName="pull" Jul 14 15:29:23.263018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262685 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3c5761-0462-4689-b1ea-968bb6c0d5de" containerName="pull" Jul 14 15:29:23.263018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262701 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3c5761-0462-4689-b1ea-968bb6c0d5de" containerName="util" Jul 14 15:29:23.263018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262710 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3c5761-0462-4689-b1ea-968bb6c0d5de" containerName="util" Jul 14 15:29:23.263018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262724 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc600c37-b36e-4d51-9160-f91d1e881813" containerName="util" Jul 14 15:29:23.263018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262733 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc600c37-b36e-4d51-9160-f91d1e881813" containerName="util" Jul 14 15:29:23.263018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262742 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc600c37-b36e-4d51-9160-f91d1e881813" containerName="pull" Jul 14 15:29:23.263018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262750 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc600c37-b36e-4d51-9160-f91d1e881813" containerName="pull" Jul 14 15:29:23.263018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262761 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3c5761-0462-4689-b1ea-968bb6c0d5de" containerName="extract" Jul 14 15:29:23.263018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262770 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3c5761-0462-4689-b1ea-968bb6c0d5de" containerName="extract" Jul 14 15:29:23.263018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262780 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83061462-6051-466b-9e61-90164f652346" containerName="extract" Jul 14 15:29:23.263018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262789 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="83061462-6051-466b-9e61-90164f652346" containerName="extract" Jul 14 15:29:23.263018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262801 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83061462-6051-466b-9e61-90164f652346" containerName="util" Jul 14 15:29:23.263018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262810 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="83061462-6051-466b-9e61-90164f652346" containerName="util" Jul 14 15:29:23.263018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262826 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83061462-6051-466b-9e61-90164f652346" containerName="pull" Jul 14 15:29:23.263018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262834 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="83061462-6051-466b-9e61-90164f652346" containerName="pull" Jul 14 15:29:23.263018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262917 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc600c37-b36e-4d51-9160-f91d1e881813" containerName="extract" Jul 14 15:29:23.263018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262928 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3c5761-0462-4689-b1ea-968bb6c0d5de" containerName="extract" Jul 14 15:29:23.263018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262942 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb58ab6d-8105-41a1-9c2f-6d74e1010daa" containerName="extract" Jul 14 15:29:23.263018 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.262951 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="83061462-6051-466b-9e61-90164f652346" containerName="extract" Jul 14 15:29:23.266187 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.266154 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-28mc5" Jul 14 15:29:23.269021 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.268990 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 14 15:29:23.269021 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.269018 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 15:29:23.270912 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.270892 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-t6r2p\"" Jul 14 15:29:23.271030 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.270942 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 15:29:23.274649 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.274611 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:29:23.302309 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.302273 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:29:23.340081 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.340046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs7m4\" (UniqueName: \"kubernetes.io/projected/0a248343-3e20-4dbf-bac9-d55c09d61474-kube-api-access-gs7m4\") pod \"limitador-limitador-54d6d988d5-28mc5\" (UID: \"0a248343-3e20-4dbf-bac9-d55c09d61474\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-28mc5" Jul 14 15:29:23.340285 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.340088 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0a248343-3e20-4dbf-bac9-d55c09d61474-config-file\") pod \"limitador-limitador-54d6d988d5-28mc5\" (UID: \"0a248343-3e20-4dbf-bac9-d55c09d61474\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-28mc5" Jul 14 15:29:23.441438 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.441395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs7m4\" (UniqueName: \"kubernetes.io/projected/0a248343-3e20-4dbf-bac9-d55c09d61474-kube-api-access-gs7m4\") pod \"limitador-limitador-54d6d988d5-28mc5\" (UID: \"0a248343-3e20-4dbf-bac9-d55c09d61474\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-28mc5" Jul 14 15:29:23.441648 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.441448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0a248343-3e20-4dbf-bac9-d55c09d61474-config-file\") pod \"limitador-limitador-54d6d988d5-28mc5\" (UID: \"0a248343-3e20-4dbf-bac9-d55c09d61474\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-28mc5" Jul 14 15:29:23.442033 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.442013 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0a248343-3e20-4dbf-bac9-d55c09d61474-config-file\") pod \"limitador-limitador-54d6d988d5-28mc5\" (UID: \"0a248343-3e20-4dbf-bac9-d55c09d61474\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-28mc5" Jul 14 15:29:23.450928 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.450904 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs7m4\" (UniqueName: \"kubernetes.io/projected/0a248343-3e20-4dbf-bac9-d55c09d61474-kube-api-access-gs7m4\") pod \"limitador-limitador-54d6d988d5-28mc5\" (UID: \"0a248343-3e20-4dbf-bac9-d55c09d61474\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-28mc5" Jul 14 15:29:23.584432 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.584384 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-28mc5" Jul 14 15:29:23.721692 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.721667 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:29:23.723986 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:29:23.723960 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a248343_3e20_4dbf_bac9_d55c09d61474.slice/crio-04ea9b500298869713e1e8b9b6598f6fc01fe726a1570bcddaeb578cc6933d3b WatchSource:0}: Error finding container 04ea9b500298869713e1e8b9b6598f6fc01fe726a1570bcddaeb578cc6933d3b: Status 404 returned error can't find the container with id 04ea9b500298869713e1e8b9b6598f6fc01fe726a1570bcddaeb578cc6933d3b Jul 14 15:29:23.738921 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.738896 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-bb4bp"] Jul 14 15:29:23.743360 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.743336 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-bb4bp" Jul 14 15:29:23.746865 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.746804 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4c294\"" Jul 14 15:29:23.749288 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.749269 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-bb4bp"] Jul 14 15:29:23.773245 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.773197 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:23.844812 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.844729 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2nsv\" (UniqueName: \"kubernetes.io/projected/66318109-6add-4db2-9558-aadf7057b268-kube-api-access-x2nsv\") pod \"authorino-58cd955675-bb4bp\" (UID: \"66318109-6add-4db2-9558-aadf7057b268\") " pod="kuadrant-system/authorino-58cd955675-bb4bp" Jul 14 15:29:23.943037 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.943002 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-kqqqd"] Jul 14 15:29:23.945317 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.945291 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2nsv\" (UniqueName: \"kubernetes.io/projected/66318109-6add-4db2-9558-aadf7057b268-kube-api-access-x2nsv\") pod \"authorino-58cd955675-bb4bp\" (UID: \"66318109-6add-4db2-9558-aadf7057b268\") " pod="kuadrant-system/authorino-58cd955675-bb4bp" Jul 14 15:29:23.946334 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.946316 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-kqqqd" Jul 14 15:29:23.953657 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.953616 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kqqqd"] Jul 14 15:29:23.956682 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:23.956659 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2nsv\" (UniqueName: \"kubernetes.io/projected/66318109-6add-4db2-9558-aadf7057b268-kube-api-access-x2nsv\") pod \"authorino-58cd955675-bb4bp\" (UID: \"66318109-6add-4db2-9558-aadf7057b268\") " pod="kuadrant-system/authorino-58cd955675-bb4bp" Jul 14 15:29:24.045748 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:24.045712 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xxhv\" (UniqueName: \"kubernetes.io/projected/bfb21244-8b94-4e0c-9de9-27ef600c1b72-kube-api-access-4xxhv\") pod \"authorino-75d556bd77-kqqqd\" (UID: \"bfb21244-8b94-4e0c-9de9-27ef600c1b72\") " pod="kuadrant-system/authorino-75d556bd77-kqqqd" Jul 14 15:29:24.053568 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:24.053543 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-bb4bp" Jul 14 15:29:24.110007 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:24.109970 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-28mc5" event={"ID":"0a248343-3e20-4dbf-bac9-d55c09d61474","Type":"ContainerStarted","Data":"04ea9b500298869713e1e8b9b6598f6fc01fe726a1570bcddaeb578cc6933d3b"} Jul 14 15:29:24.146863 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:24.146817 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xxhv\" (UniqueName: \"kubernetes.io/projected/bfb21244-8b94-4e0c-9de9-27ef600c1b72-kube-api-access-4xxhv\") pod \"authorino-75d556bd77-kqqqd\" (UID: \"bfb21244-8b94-4e0c-9de9-27ef600c1b72\") " pod="kuadrant-system/authorino-75d556bd77-kqqqd" Jul 14 15:29:24.156698 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:24.156664 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xxhv\" (UniqueName: \"kubernetes.io/projected/bfb21244-8b94-4e0c-9de9-27ef600c1b72-kube-api-access-4xxhv\") pod \"authorino-75d556bd77-kqqqd\" (UID: \"bfb21244-8b94-4e0c-9de9-27ef600c1b72\") " pod="kuadrant-system/authorino-75d556bd77-kqqqd" Jul 14 15:29:24.186525 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:24.186491 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-bb4bp"] Jul 14 15:29:24.188214 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:29:24.188185 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66318109_6add_4db2_9558_aadf7057b268.slice/crio-af444ab508e782945b9595e9f5bda02e931e960736f51b8642298c61d98f6b44 WatchSource:0}: Error finding container af444ab508e782945b9595e9f5bda02e931e960736f51b8642298c61d98f6b44: Status 404 returned error can't find the container with id af444ab508e782945b9595e9f5bda02e931e960736f51b8642298c61d98f6b44 Jul 14 15:29:24.267057 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:24.267026 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-kqqqd" Jul 14 15:29:24.437874 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:24.437846 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kqqqd"] Jul 14 15:29:24.439794 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:29:24.439751 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfb21244_8b94_4e0c_9de9_27ef600c1b72.slice/crio-4bf83a36c30b0484681986fe6c39c51aff86bccd883d05080c19eb95b269e1f8 WatchSource:0}: Error finding container 4bf83a36c30b0484681986fe6c39c51aff86bccd883d05080c19eb95b269e1f8: Status 404 returned error can't find the container with id 4bf83a36c30b0484681986fe6c39c51aff86bccd883d05080c19eb95b269e1f8 Jul 14 15:29:25.126092 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:25.126042 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-kqqqd" event={"ID":"bfb21244-8b94-4e0c-9de9-27ef600c1b72","Type":"ContainerStarted","Data":"4bf83a36c30b0484681986fe6c39c51aff86bccd883d05080c19eb95b269e1f8"} Jul 14 15:29:25.130907 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:25.130876 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-bb4bp" event={"ID":"66318109-6add-4db2-9558-aadf7057b268","Type":"ContainerStarted","Data":"af444ab508e782945b9595e9f5bda02e931e960736f51b8642298c61d98f6b44"} Jul 14 15:29:26.397334 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:26.397286 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:26.450961 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:26.450917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:29.166658 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:29.166555 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-bb4bp" event={"ID":"66318109-6add-4db2-9558-aadf7057b268","Type":"ContainerStarted","Data":"3a82c5b35ad0fcff5091332fa1bb87bc8a2f4cdff753705a39339a9af3f5ebe7"} Jul 14 15:29:29.168088 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:29.168065 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-28mc5" event={"ID":"0a248343-3e20-4dbf-bac9-d55c09d61474","Type":"ContainerStarted","Data":"2c95490c4e9d7da2a128f5652543091a5b00f9cce3fc9a8898731427bceaf3a8"} Jul 14 15:29:29.168184 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:29.168113 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-28mc5" Jul 14 15:29:29.169421 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:29.169398 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-kqqqd" event={"ID":"bfb21244-8b94-4e0c-9de9-27ef600c1b72","Type":"ContainerStarted","Data":"0cf8977bd246c26ba3d1715ea602068e857709421075a1ce887811daaf63f2f0"} Jul 14 15:29:29.183899 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:29.183858 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-bb4bp" podStartSLOduration=1.517846457 podStartE2EDuration="6.183845712s" totalImagesPullingTime="4.665999255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:29:29.182871502 +0000 UTC m=+510.035254674" watchObservedRunningTime="2026-07-14 15:29:29.183845712 +0000 UTC m=+510.036228875" Jul 14 15:29:29.200084 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:29.200032 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-kqqqd" podStartSLOduration=1.773977628 podStartE2EDuration="6.200013639s" totalImagesPullingTime="4.426036011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:29:29.198788026 +0000 UTC m=+510.051171191" watchObservedRunningTime="2026-07-14 15:29:29.200013639 +0000 UTC m=+510.052396805" Jul 14 15:29:29.217590 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:29.217543 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-28mc5" podStartSLOduration=1.087429947 podStartE2EDuration="6.21752754s" totalImagesPullingTime="5.130097593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:29:29.216227431 +0000 UTC m=+510.068610595" watchObservedRunningTime="2026-07-14 15:29:29.21752754 +0000 UTC m=+510.069910705" Jul 14 15:29:29.230911 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:29.230884 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-bb4bp"] Jul 14 15:29:31.180023 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:31.179978 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-bb4bp" podUID="66318109-6add-4db2-9558-aadf7057b268" containerName="authorino" containerID="cri-o://3a82c5b35ad0fcff5091332fa1bb87bc8a2f4cdff753705a39339a9af3f5ebe7" gracePeriod=30 Jul 14 15:29:31.432307 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:31.432254 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-bb4bp" Jul 14 15:29:31.516640 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:31.516604 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2nsv\" (UniqueName: \"kubernetes.io/projected/66318109-6add-4db2-9558-aadf7057b268-kube-api-access-x2nsv\") pod \"66318109-6add-4db2-9558-aadf7057b268\" (UID: \"66318109-6add-4db2-9558-aadf7057b268\") " Jul 14 15:29:31.518805 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:31.518772 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66318109-6add-4db2-9558-aadf7057b268-kube-api-access-x2nsv" (OuterVolumeSpecName: "kube-api-access-x2nsv") pod "66318109-6add-4db2-9558-aadf7057b268" (UID: "66318109-6add-4db2-9558-aadf7057b268"). InnerVolumeSpecName "kube-api-access-x2nsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:29:31.617755 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:31.617720 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2nsv\" (UniqueName: \"kubernetes.io/projected/66318109-6add-4db2-9558-aadf7057b268-kube-api-access-x2nsv\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:29:32.186091 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:32.186039 2579 generic.go:356] "Generic (PLEG): container finished" podID="66318109-6add-4db2-9558-aadf7057b268" containerID="3a82c5b35ad0fcff5091332fa1bb87bc8a2f4cdff753705a39339a9af3f5ebe7" exitCode=0 Jul 14 15:29:32.186617 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:32.186109 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-bb4bp" Jul 14 15:29:32.186617 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:32.186127 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-bb4bp" event={"ID":"66318109-6add-4db2-9558-aadf7057b268","Type":"ContainerDied","Data":"3a82c5b35ad0fcff5091332fa1bb87bc8a2f4cdff753705a39339a9af3f5ebe7"} Jul 14 15:29:32.186617 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:32.186194 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-bb4bp" event={"ID":"66318109-6add-4db2-9558-aadf7057b268","Type":"ContainerDied","Data":"af444ab508e782945b9595e9f5bda02e931e960736f51b8642298c61d98f6b44"} Jul 14 15:29:32.186617 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:32.186256 2579 scope.go:117] "RemoveContainer" containerID="3a82c5b35ad0fcff5091332fa1bb87bc8a2f4cdff753705a39339a9af3f5ebe7" Jul 14 15:29:32.194993 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:32.194975 2579 scope.go:117] "RemoveContainer" containerID="3a82c5b35ad0fcff5091332fa1bb87bc8a2f4cdff753705a39339a9af3f5ebe7" Jul 14 15:29:32.195298 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:29:32.195275 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a82c5b35ad0fcff5091332fa1bb87bc8a2f4cdff753705a39339a9af3f5ebe7\": container with ID starting with 3a82c5b35ad0fcff5091332fa1bb87bc8a2f4cdff753705a39339a9af3f5ebe7 not found: ID does not exist" containerID="3a82c5b35ad0fcff5091332fa1bb87bc8a2f4cdff753705a39339a9af3f5ebe7" Jul 14 15:29:32.195370 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:32.195310 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a82c5b35ad0fcff5091332fa1bb87bc8a2f4cdff753705a39339a9af3f5ebe7"} err="failed to get container status \"3a82c5b35ad0fcff5091332fa1bb87bc8a2f4cdff753705a39339a9af3f5ebe7\": rpc error: code = NotFound desc = could not find container \"3a82c5b35ad0fcff5091332fa1bb87bc8a2f4cdff753705a39339a9af3f5ebe7\": container with ID starting with 3a82c5b35ad0fcff5091332fa1bb87bc8a2f4cdff753705a39339a9af3f5ebe7 not found: ID does not exist" Jul 14 15:29:32.223414 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:32.223381 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-bb4bp"] Jul 14 15:29:32.243545 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:32.243516 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-bb4bp"] Jul 14 15:29:33.697105 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:33.697073 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66318109-6add-4db2-9558-aadf7057b268" path="/var/lib/kubelet/pods/66318109-6add-4db2-9558-aadf7057b268/volumes" Jul 14 15:29:40.175928 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:40.175898 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-28mc5" Jul 14 15:29:44.368147 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:44.368098 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:44.515756 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:44.515714 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:44.868374 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:44.868332 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:57.300334 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.300239 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:57.489908 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.489867 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:29:57.533188 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.533137 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tnqj2"] Jul 14 15:29:57.533496 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.533483 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66318109-6add-4db2-9558-aadf7057b268" containerName="authorino" Jul 14 15:29:57.533543 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.533498 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="66318109-6add-4db2-9558-aadf7057b268" containerName="authorino" Jul 14 15:29:57.533587 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.533567 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="66318109-6add-4db2-9558-aadf7057b268" containerName="authorino" Jul 14 15:29:57.536700 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.536683 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tnqj2" Jul 14 15:29:57.541312 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.541284 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tnqj2"] Jul 14 15:29:57.544798 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.544757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7vph\" (UniqueName: \"kubernetes.io/projected/3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377-kube-api-access-v7vph\") pod \"authorino-5d4b67c446-tnqj2\" (UID: \"3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377\") " pod="kuadrant-system/authorino-5d4b67c446-tnqj2" Jul 14 15:29:57.646328 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.646286 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7vph\" (UniqueName: \"kubernetes.io/projected/3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377-kube-api-access-v7vph\") pod \"authorino-5d4b67c446-tnqj2\" (UID: \"3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377\") " pod="kuadrant-system/authorino-5d4b67c446-tnqj2" Jul 14 15:29:57.656100 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.656061 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7vph\" (UniqueName: \"kubernetes.io/projected/3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377-kube-api-access-v7vph\") pod \"authorino-5d4b67c446-tnqj2\" (UID: \"3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377\") " pod="kuadrant-system/authorino-5d4b67c446-tnqj2" Jul 14 15:29:57.754766 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.754728 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tnqj2"] Jul 14 15:29:57.754997 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.754985 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tnqj2" Jul 14 15:29:57.783037 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.783005 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-sf8b9"] Jul 14 15:29:57.787483 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.787460 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-sf8b9" Jul 14 15:29:57.790412 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.790384 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 14 15:29:57.793543 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.793520 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-sf8b9"] Jul 14 15:29:57.832637 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.832603 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-sf8b9"] Jul 14 15:29:57.832916 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:29:57.832893 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5zsbg tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-5zsbg tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-sf8b9" podUID="99499eab-690b-4850-8470-42662e0ed574" Jul 14 15:29:57.847947 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.847906 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99499eab-690b-4850-8470-42662e0ed574-tls-cert\") pod \"authorino-76877994-sf8b9\" (UID: \"99499eab-690b-4850-8470-42662e0ed574\") " pod="kuadrant-system/authorino-76877994-sf8b9" Jul 14 15:29:57.848154 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.847975 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zsbg\" (UniqueName: \"kubernetes.io/projected/99499eab-690b-4850-8470-42662e0ed574-kube-api-access-5zsbg\") pod \"authorino-76877994-sf8b9\" (UID: \"99499eab-690b-4850-8470-42662e0ed574\") " pod="kuadrant-system/authorino-76877994-sf8b9" Jul 14 15:29:57.857463 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.857430 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57445f78c8-js2hn"] Jul 14 15:29:57.861051 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.861027 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57445f78c8-js2hn" Jul 14 15:29:57.870399 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.869586 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57445f78c8-js2hn"] Jul 14 15:29:57.913311 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.913278 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tnqj2"] Jul 14 15:29:57.915087 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:29:57.915059 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f8c61cc_980a_48d8_8d4b_3dd3f0e0d377.slice/crio-cc2c76b53ddb408676a1f8aa83d2f1fc000f26eff1262a7c07e623d013316623 WatchSource:0}: Error finding container cc2c76b53ddb408676a1f8aa83d2f1fc000f26eff1262a7c07e623d013316623: Status 404 returned error can't find the container with id cc2c76b53ddb408676a1f8aa83d2f1fc000f26eff1262a7c07e623d013316623 Jul 14 15:29:57.949356 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.949317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99499eab-690b-4850-8470-42662e0ed574-tls-cert\") pod \"authorino-76877994-sf8b9\" (UID: \"99499eab-690b-4850-8470-42662e0ed574\") " pod="kuadrant-system/authorino-76877994-sf8b9" Jul 14 15:29:57.949356 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.949360 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zsbg\" (UniqueName: \"kubernetes.io/projected/99499eab-690b-4850-8470-42662e0ed574-kube-api-access-5zsbg\") pod \"authorino-76877994-sf8b9\" (UID: \"99499eab-690b-4850-8470-42662e0ed574\") " pod="kuadrant-system/authorino-76877994-sf8b9" Jul 14 15:29:57.949581 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.949422 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z95f7\" (UniqueName: \"kubernetes.io/projected/95b4264d-ae58-4f6a-9046-ad9eecf8db86-kube-api-access-z95f7\") pod \"authorino-57445f78c8-js2hn\" (UID: \"95b4264d-ae58-4f6a-9046-ad9eecf8db86\") " pod="kuadrant-system/authorino-57445f78c8-js2hn" Jul 14 15:29:57.949581 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.949457 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/95b4264d-ae58-4f6a-9046-ad9eecf8db86-tls-cert\") pod \"authorino-57445f78c8-js2hn\" (UID: \"95b4264d-ae58-4f6a-9046-ad9eecf8db86\") " pod="kuadrant-system/authorino-57445f78c8-js2hn" Jul 14 15:29:57.952048 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.952023 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99499eab-690b-4850-8470-42662e0ed574-tls-cert\") pod \"authorino-76877994-sf8b9\" (UID: \"99499eab-690b-4850-8470-42662e0ed574\") " pod="kuadrant-system/authorino-76877994-sf8b9" Jul 14 15:29:57.961385 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:57.961360 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zsbg\" (UniqueName: \"kubernetes.io/projected/99499eab-690b-4850-8470-42662e0ed574-kube-api-access-5zsbg\") pod \"authorino-76877994-sf8b9\" (UID: \"99499eab-690b-4850-8470-42662e0ed574\") " pod="kuadrant-system/authorino-76877994-sf8b9" Jul 14 15:29:58.050442 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:58.050402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z95f7\" (UniqueName: \"kubernetes.io/projected/95b4264d-ae58-4f6a-9046-ad9eecf8db86-kube-api-access-z95f7\") pod \"authorino-57445f78c8-js2hn\" (UID: \"95b4264d-ae58-4f6a-9046-ad9eecf8db86\") " pod="kuadrant-system/authorino-57445f78c8-js2hn" Jul 14 15:29:58.050626 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:58.050456 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/95b4264d-ae58-4f6a-9046-ad9eecf8db86-tls-cert\") pod \"authorino-57445f78c8-js2hn\" (UID: \"95b4264d-ae58-4f6a-9046-ad9eecf8db86\") " pod="kuadrant-system/authorino-57445f78c8-js2hn" Jul 14 15:29:58.052992 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:58.052971 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/95b4264d-ae58-4f6a-9046-ad9eecf8db86-tls-cert\") pod \"authorino-57445f78c8-js2hn\" (UID: \"95b4264d-ae58-4f6a-9046-ad9eecf8db86\") " pod="kuadrant-system/authorino-57445f78c8-js2hn" Jul 14 15:29:58.060367 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:58.060343 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z95f7\" (UniqueName: \"kubernetes.io/projected/95b4264d-ae58-4f6a-9046-ad9eecf8db86-kube-api-access-z95f7\") pod \"authorino-57445f78c8-js2hn\" (UID: \"95b4264d-ae58-4f6a-9046-ad9eecf8db86\") " pod="kuadrant-system/authorino-57445f78c8-js2hn" Jul 14 15:29:58.174865 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:58.174767 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57445f78c8-js2hn" Jul 14 15:29:58.312205 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:58.312157 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57445f78c8-js2hn"] Jul 14 15:29:58.313903 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:29:58.313874 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95b4264d_ae58_4f6a_9046_ad9eecf8db86.slice/crio-b31d63c1a6d75950d0842326605e40a6eb2ee5b147774d101c9089b98f15826a WatchSource:0}: Error finding container b31d63c1a6d75950d0842326605e40a6eb2ee5b147774d101c9089b98f15826a: Status 404 returned error can't find the container with id b31d63c1a6d75950d0842326605e40a6eb2ee5b147774d101c9089b98f15826a Jul 14 15:29:58.318820 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:58.318786 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-tnqj2" event={"ID":"3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377","Type":"ContainerStarted","Data":"cc2c76b53ddb408676a1f8aa83d2f1fc000f26eff1262a7c07e623d013316623"} Jul 14 15:29:58.318900 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:58.318802 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-sf8b9" Jul 14 15:29:58.326137 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:58.326117 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-sf8b9" Jul 14 15:29:58.354474 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:58.354449 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99499eab-690b-4850-8470-42662e0ed574-tls-cert\") pod \"99499eab-690b-4850-8470-42662e0ed574\" (UID: \"99499eab-690b-4850-8470-42662e0ed574\") " Jul 14 15:29:58.354591 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:58.354501 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zsbg\" (UniqueName: \"kubernetes.io/projected/99499eab-690b-4850-8470-42662e0ed574-kube-api-access-5zsbg\") pod \"99499eab-690b-4850-8470-42662e0ed574\" (UID: \"99499eab-690b-4850-8470-42662e0ed574\") " Jul 14 15:29:58.356781 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:58.356750 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99499eab-690b-4850-8470-42662e0ed574-kube-api-access-5zsbg" (OuterVolumeSpecName: "kube-api-access-5zsbg") pod "99499eab-690b-4850-8470-42662e0ed574" (UID: "99499eab-690b-4850-8470-42662e0ed574"). InnerVolumeSpecName "kube-api-access-5zsbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:29:58.356870 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:58.356831 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99499eab-690b-4850-8470-42662e0ed574-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "99499eab-690b-4850-8470-42662e0ed574" (UID: "99499eab-690b-4850-8470-42662e0ed574"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 15:29:58.455148 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:58.455054 2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99499eab-690b-4850-8470-42662e0ed574-tls-cert\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:29:58.455148 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:58.455087 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zsbg\" (UniqueName: \"kubernetes.io/projected/99499eab-690b-4850-8470-42662e0ed574-kube-api-access-5zsbg\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:29:59.324996 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:59.324955 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-tnqj2" event={"ID":"3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377","Type":"ContainerStarted","Data":"a68de8d1856b45794458fc7c00e836f2fb7f00233bcbb37ebf1ac5d50d5539a0"} Jul 14 15:29:59.324996 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:59.324986 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-tnqj2" podUID="3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377" containerName="authorino" containerID="cri-o://a68de8d1856b45794458fc7c00e836f2fb7f00233bcbb37ebf1ac5d50d5539a0" gracePeriod=30 Jul 14 15:29:59.326616 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:59.326597 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-sf8b9" Jul 14 15:29:59.326744 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:59.326596 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57445f78c8-js2hn" event={"ID":"95b4264d-ae58-4f6a-9046-ad9eecf8db86","Type":"ContainerStarted","Data":"7766650c407b0005af923ee2d3747a9c411c1e66ddb9757757dc00d738b25f3e"} Jul 14 15:29:59.326744 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:59.326714 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57445f78c8-js2hn" event={"ID":"95b4264d-ae58-4f6a-9046-ad9eecf8db86","Type":"ContainerStarted","Data":"b31d63c1a6d75950d0842326605e40a6eb2ee5b147774d101c9089b98f15826a"} Jul 14 15:29:59.344609 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:59.344563 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-tnqj2" podStartSLOduration=1.916387881 podStartE2EDuration="2.344546824s" totalImagesPullingTime="428.158943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:29:59.343606791 +0000 UTC m=+540.195989957" watchObservedRunningTime="2026-07-14 15:29:59.344546824 +0000 UTC m=+540.196929988" Jul 14 15:29:59.373788 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:59.373758 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-sf8b9"] Jul 14 15:29:59.375762 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:59.375738 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-sf8b9"] Jul 14 15:29:59.390153 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:59.390098 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-57445f78c8-js2hn" podStartSLOduration=1.994014822 podStartE2EDuration="2.390081555s" totalImagesPullingTime="396.066733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:29:59.388644912 +0000 UTC m=+540.241028078" watchObservedRunningTime="2026-07-14 15:29:59.390081555 +0000 UTC m=+540.242464698" Jul 14 15:29:59.423327 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:59.423290 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kqqqd"] Jul 14 15:29:59.423514 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:59.423483 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-kqqqd" podUID="bfb21244-8b94-4e0c-9de9-27ef600c1b72" containerName="authorino" containerID="cri-o://0cf8977bd246c26ba3d1715ea602068e857709421075a1ce887811daaf63f2f0" gracePeriod=30 Jul 14 15:29:59.584636 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:59.584565 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tnqj2" Jul 14 15:29:59.662534 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:59.662504 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7vph\" (UniqueName: \"kubernetes.io/projected/3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377-kube-api-access-v7vph\") pod \"3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377\" (UID: \"3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377\") " Jul 14 15:29:59.665325 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:59.665284 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377-kube-api-access-v7vph" (OuterVolumeSpecName: "kube-api-access-v7vph") pod "3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377" (UID: "3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377"). InnerVolumeSpecName "kube-api-access-v7vph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:29:59.698327 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:59.698299 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99499eab-690b-4850-8470-42662e0ed574" path="/var/lib/kubelet/pods/99499eab-690b-4850-8470-42662e0ed574/volumes" Jul 14 15:29:59.731373 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:59.731343 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-kqqqd" Jul 14 15:29:59.763009 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:59.762974 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xxhv\" (UniqueName: \"kubernetes.io/projected/bfb21244-8b94-4e0c-9de9-27ef600c1b72-kube-api-access-4xxhv\") pod \"bfb21244-8b94-4e0c-9de9-27ef600c1b72\" (UID: \"bfb21244-8b94-4e0c-9de9-27ef600c1b72\") " Jul 14 15:29:59.763248 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:59.763229 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7vph\" (UniqueName: \"kubernetes.io/projected/3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377-kube-api-access-v7vph\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:29:59.765662 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:59.765629 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb21244-8b94-4e0c-9de9-27ef600c1b72-kube-api-access-4xxhv" (OuterVolumeSpecName: "kube-api-access-4xxhv") pod "bfb21244-8b94-4e0c-9de9-27ef600c1b72" (UID: "bfb21244-8b94-4e0c-9de9-27ef600c1b72"). InnerVolumeSpecName "kube-api-access-4xxhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:29:59.864496 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:29:59.864413 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xxhv\" (UniqueName: \"kubernetes.io/projected/bfb21244-8b94-4e0c-9de9-27ef600c1b72-kube-api-access-4xxhv\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:30:00.332647 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:00.332615 2579 generic.go:356] "Generic (PLEG): container finished" podID="bfb21244-8b94-4e0c-9de9-27ef600c1b72" containerID="0cf8977bd246c26ba3d1715ea602068e857709421075a1ce887811daaf63f2f0" exitCode=0 Jul 14 15:30:00.333081 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:00.332659 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-kqqqd" Jul 14 15:30:00.333081 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:00.332712 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-kqqqd" event={"ID":"bfb21244-8b94-4e0c-9de9-27ef600c1b72","Type":"ContainerDied","Data":"0cf8977bd246c26ba3d1715ea602068e857709421075a1ce887811daaf63f2f0"} Jul 14 15:30:00.333081 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:00.332758 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-kqqqd" event={"ID":"bfb21244-8b94-4e0c-9de9-27ef600c1b72","Type":"ContainerDied","Data":"4bf83a36c30b0484681986fe6c39c51aff86bccd883d05080c19eb95b269e1f8"} Jul 14 15:30:00.333081 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:00.332780 2579 scope.go:117] "RemoveContainer" containerID="0cf8977bd246c26ba3d1715ea602068e857709421075a1ce887811daaf63f2f0" Jul 14 15:30:00.334487 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:00.334114 2579 generic.go:356] "Generic (PLEG): container finished" podID="3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377" containerID="a68de8d1856b45794458fc7c00e836f2fb7f00233bcbb37ebf1ac5d50d5539a0" exitCode=0 Jul 14 15:30:00.334487 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:00.334149 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tnqj2" Jul 14 15:30:00.334487 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:00.334197 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-tnqj2" event={"ID":"3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377","Type":"ContainerDied","Data":"a68de8d1856b45794458fc7c00e836f2fb7f00233bcbb37ebf1ac5d50d5539a0"} Jul 14 15:30:00.334487 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:00.334238 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-tnqj2" event={"ID":"3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377","Type":"ContainerDied","Data":"cc2c76b53ddb408676a1f8aa83d2f1fc000f26eff1262a7c07e623d013316623"} Jul 14 15:30:00.342721 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:00.342702 2579 scope.go:117] "RemoveContainer" containerID="0cf8977bd246c26ba3d1715ea602068e857709421075a1ce887811daaf63f2f0" Jul 14 15:30:00.343014 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:30:00.342994 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf8977bd246c26ba3d1715ea602068e857709421075a1ce887811daaf63f2f0\": container with ID starting with 0cf8977bd246c26ba3d1715ea602068e857709421075a1ce887811daaf63f2f0 not found: ID does not exist" containerID="0cf8977bd246c26ba3d1715ea602068e857709421075a1ce887811daaf63f2f0" Jul 14 15:30:00.343082 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:00.343024 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf8977bd246c26ba3d1715ea602068e857709421075a1ce887811daaf63f2f0"} err="failed to get container status \"0cf8977bd246c26ba3d1715ea602068e857709421075a1ce887811daaf63f2f0\": rpc error: code = NotFound desc = could not find container \"0cf8977bd246c26ba3d1715ea602068e857709421075a1ce887811daaf63f2f0\": container with ID starting with 0cf8977bd246c26ba3d1715ea602068e857709421075a1ce887811daaf63f2f0 not found: ID does not exist" Jul 14 15:30:00.343082 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:00.343040 2579 scope.go:117] "RemoveContainer" containerID="a68de8d1856b45794458fc7c00e836f2fb7f00233bcbb37ebf1ac5d50d5539a0" Jul 14 15:30:00.351074 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:00.351050 2579 scope.go:117] "RemoveContainer" containerID="a68de8d1856b45794458fc7c00e836f2fb7f00233bcbb37ebf1ac5d50d5539a0" Jul 14 15:30:00.351381 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:30:00.351365 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a68de8d1856b45794458fc7c00e836f2fb7f00233bcbb37ebf1ac5d50d5539a0\": container with ID starting with a68de8d1856b45794458fc7c00e836f2fb7f00233bcbb37ebf1ac5d50d5539a0 not found: ID does not exist" containerID="a68de8d1856b45794458fc7c00e836f2fb7f00233bcbb37ebf1ac5d50d5539a0" Jul 14 15:30:00.351440 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:00.351387 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a68de8d1856b45794458fc7c00e836f2fb7f00233bcbb37ebf1ac5d50d5539a0"} err="failed to get container status \"a68de8d1856b45794458fc7c00e836f2fb7f00233bcbb37ebf1ac5d50d5539a0\": rpc error: code = NotFound desc = could not find container \"a68de8d1856b45794458fc7c00e836f2fb7f00233bcbb37ebf1ac5d50d5539a0\": container with ID starting with a68de8d1856b45794458fc7c00e836f2fb7f00233bcbb37ebf1ac5d50d5539a0 not found: ID does not exist" Jul 14 15:30:00.354981 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:00.354954 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tnqj2"] Jul 14 15:30:00.361614 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:00.361588 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tnqj2"] Jul 14 15:30:00.375719 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:00.375688 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kqqqd"] Jul 14 15:30:00.379414 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:00.379390 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kqqqd"] Jul 14 15:30:01.696763 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:01.696724 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377" path="/var/lib/kubelet/pods/3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377/volumes" Jul 14 15:30:01.697135 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:01.697038 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfb21244-8b94-4e0c-9de9-27ef600c1b72" path="/var/lib/kubelet/pods/bfb21244-8b94-4e0c-9de9-27ef600c1b72/volumes" Jul 14 15:30:04.408682 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:04.408649 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:30:04.506081 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:04.506030 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:30:04.570871 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:04.570837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:30:04.783279 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:04.783163 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:30:08.723065 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:08.723027 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:30:08.825214 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:08.825161 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:30:41.386615 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:41.386580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:30:41.484803 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:41.484752 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:30:42.038990 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:42.038953 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:30:42.292624 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:42.292533 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:30:42.417389 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:42.417359 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:30:42.515358 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:42.515325 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:30:42.562016 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:42.561944 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:30:53.265239 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:53.265165 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:30:53.362727 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:53.362688 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:30:59.613600 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:59.613572 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 15:30:59.622513 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:30:59.622490 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 15:31:04.513923 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:31:04.513889 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:31:04.980055 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:31:04.980016 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:31:15.297927 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:31:15.297881 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:31:16.081271 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:31:16.081229 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:31:30.516343 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:31:30.516302 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:31:31.675515 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:31:31.675478 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:31:32.164965 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:31:32.164924 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:31:32.265794 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:31:32.265757 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:31:35.298442 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:31:35.298394 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:31:36.675909 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:31:36.675863 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:31:37.838887 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:31:37.838845 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:31:48.937160 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:31:48.937120 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:31:49.620165 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:31:49.620133 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:31:51.980984 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:31:51.980947 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:31:56.800555 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:31:56.800518 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:31:59.677233 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:31:59.677198 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:32:13.332471 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:32:13.332428 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:32:27.768707 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:32:27.768670 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:32:31.972306 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:32:31.972265 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:33:02.725256 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:02.725156 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f4464dd95-gw85n"] Jul 14 15:33:02.725750 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:02.725486 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb21244-8b94-4e0c-9de9-27ef600c1b72" containerName="authorino" Jul 14 15:33:02.725750 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:02.725497 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb21244-8b94-4e0c-9de9-27ef600c1b72" containerName="authorino" Jul 14 15:33:02.725750 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:02.725515 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377" containerName="authorino" Jul 14 15:33:02.725750 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:02.725521 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377" containerName="authorino" Jul 14 15:33:02.725750 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:02.725594 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f8c61cc-980a-48d8-8d4b-3dd3f0e0d377" containerName="authorino" Jul 14 15:33:02.725750 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:02.725604 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb21244-8b94-4e0c-9de9-27ef600c1b72" containerName="authorino" Jul 14 15:33:02.727449 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:02.727433 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f4464dd95-gw85n" Jul 14 15:33:02.736331 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:02.736303 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f4464dd95-gw85n"] Jul 14 15:33:02.796251 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:02.796225 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9548884b-0ad9-404a-a7bb-cd8dd672957d-tls-cert\") pod \"authorino-6f4464dd95-gw85n\" (UID: \"9548884b-0ad9-404a-a7bb-cd8dd672957d\") " pod="kuadrant-system/authorino-6f4464dd95-gw85n" Jul 14 15:33:02.796392 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:02.796278 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqf87\" (UniqueName: \"kubernetes.io/projected/9548884b-0ad9-404a-a7bb-cd8dd672957d-kube-api-access-jqf87\") pod \"authorino-6f4464dd95-gw85n\" (UID: \"9548884b-0ad9-404a-a7bb-cd8dd672957d\") " pod="kuadrant-system/authorino-6f4464dd95-gw85n" Jul 14 15:33:02.897088 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:02.897052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqf87\" (UniqueName: \"kubernetes.io/projected/9548884b-0ad9-404a-a7bb-cd8dd672957d-kube-api-access-jqf87\") pod \"authorino-6f4464dd95-gw85n\" (UID: \"9548884b-0ad9-404a-a7bb-cd8dd672957d\") " pod="kuadrant-system/authorino-6f4464dd95-gw85n" Jul 14 15:33:02.897301 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:02.897122 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9548884b-0ad9-404a-a7bb-cd8dd672957d-tls-cert\") pod \"authorino-6f4464dd95-gw85n\" (UID: \"9548884b-0ad9-404a-a7bb-cd8dd672957d\") " pod="kuadrant-system/authorino-6f4464dd95-gw85n" Jul 14 15:33:02.899806 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:02.899774 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9548884b-0ad9-404a-a7bb-cd8dd672957d-tls-cert\") pod \"authorino-6f4464dd95-gw85n\" (UID: \"9548884b-0ad9-404a-a7bb-cd8dd672957d\") " pod="kuadrant-system/authorino-6f4464dd95-gw85n" Jul 14 15:33:02.906281 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:02.906258 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqf87\" (UniqueName: \"kubernetes.io/projected/9548884b-0ad9-404a-a7bb-cd8dd672957d-kube-api-access-jqf87\") pod \"authorino-6f4464dd95-gw85n\" (UID: \"9548884b-0ad9-404a-a7bb-cd8dd672957d\") " pod="kuadrant-system/authorino-6f4464dd95-gw85n" Jul 14 15:33:03.037497 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:03.037426 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f4464dd95-gw85n" Jul 14 15:33:03.082701 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:03.082613 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:33:03.177514 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:03.177432 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:33:03.177779 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:03.177694 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f4464dd95-gw85n"] Jul 14 15:33:03.180761 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:33:03.180735 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9548884b_0ad9_404a_a7bb_cd8dd672957d.slice/crio-9fd3e6e2b41838e5e8cb0bac7999ab98b095fcaea588aa77d22e8d5575cba3e7 WatchSource:0}: Error finding container 9fd3e6e2b41838e5e8cb0bac7999ab98b095fcaea588aa77d22e8d5575cba3e7: Status 404 returned error can't find the container with id 9fd3e6e2b41838e5e8cb0bac7999ab98b095fcaea588aa77d22e8d5575cba3e7 Jul 14 15:33:03.182450 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:03.182433 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 15:33:03.251593 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:03.251557 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f4464dd95-gw85n" event={"ID":"9548884b-0ad9-404a-a7bb-cd8dd672957d","Type":"ContainerStarted","Data":"9fd3e6e2b41838e5e8cb0bac7999ab98b095fcaea588aa77d22e8d5575cba3e7"} Jul 14 15:33:04.260628 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:04.260571 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f4464dd95-gw85n" event={"ID":"9548884b-0ad9-404a-a7bb-cd8dd672957d","Type":"ContainerStarted","Data":"407908146c538179f1d31d4e045ba1f0f08ccec99259d4e9df5349c99b249eb7"} Jul 14 15:33:04.279708 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:04.279649 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6f4464dd95-gw85n" podStartSLOduration=1.880267235 podStartE2EDuration="2.279627232s" totalImagesPullingTime="399.359997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:33:04.276944186 +0000 UTC m=+725.129327351" watchObservedRunningTime="2026-07-14 15:33:04.279627232 +0000 UTC m=+725.132010399" Jul 14 15:33:04.313550 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:04.313512 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57445f78c8-js2hn"] Jul 14 15:33:04.314014 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:04.313979 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-57445f78c8-js2hn" podUID="95b4264d-ae58-4f6a-9046-ad9eecf8db86" containerName="authorino" containerID="cri-o://7766650c407b0005af923ee2d3747a9c411c1e66ddb9757757dc00d738b25f3e" gracePeriod=30 Jul 14 15:33:04.374493 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:04.374464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:33:04.475956 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:04.475917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:33:04.576864 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:04.576840 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57445f78c8-js2hn" Jul 14 15:33:04.616696 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:04.616664 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z95f7\" (UniqueName: \"kubernetes.io/projected/95b4264d-ae58-4f6a-9046-ad9eecf8db86-kube-api-access-z95f7\") pod \"95b4264d-ae58-4f6a-9046-ad9eecf8db86\" (UID: \"95b4264d-ae58-4f6a-9046-ad9eecf8db86\") " Jul 14 15:33:04.616903 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:04.616735 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/95b4264d-ae58-4f6a-9046-ad9eecf8db86-tls-cert\") pod \"95b4264d-ae58-4f6a-9046-ad9eecf8db86\" (UID: \"95b4264d-ae58-4f6a-9046-ad9eecf8db86\") " Jul 14 15:33:04.618944 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:04.618906 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b4264d-ae58-4f6a-9046-ad9eecf8db86-kube-api-access-z95f7" (OuterVolumeSpecName: "kube-api-access-z95f7") pod "95b4264d-ae58-4f6a-9046-ad9eecf8db86" (UID: "95b4264d-ae58-4f6a-9046-ad9eecf8db86"). InnerVolumeSpecName "kube-api-access-z95f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:33:04.627910 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:04.627881 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95b4264d-ae58-4f6a-9046-ad9eecf8db86-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "95b4264d-ae58-4f6a-9046-ad9eecf8db86" (UID: "95b4264d-ae58-4f6a-9046-ad9eecf8db86"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 15:33:04.718281 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:04.718251 2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/95b4264d-ae58-4f6a-9046-ad9eecf8db86-tls-cert\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:33:04.718281 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:04.718276 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z95f7\" (UniqueName: \"kubernetes.io/projected/95b4264d-ae58-4f6a-9046-ad9eecf8db86-kube-api-access-z95f7\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:33:05.268076 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:05.267996 2579 generic.go:356] "Generic (PLEG): container finished" podID="95b4264d-ae58-4f6a-9046-ad9eecf8db86" containerID="7766650c407b0005af923ee2d3747a9c411c1e66ddb9757757dc00d738b25f3e" exitCode=0 Jul 14 15:33:05.268076 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:05.268036 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57445f78c8-js2hn" Jul 14 15:33:05.268568 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:05.268074 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57445f78c8-js2hn" event={"ID":"95b4264d-ae58-4f6a-9046-ad9eecf8db86","Type":"ContainerDied","Data":"7766650c407b0005af923ee2d3747a9c411c1e66ddb9757757dc00d738b25f3e"} Jul 14 15:33:05.268568 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:05.268119 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57445f78c8-js2hn" event={"ID":"95b4264d-ae58-4f6a-9046-ad9eecf8db86","Type":"ContainerDied","Data":"b31d63c1a6d75950d0842326605e40a6eb2ee5b147774d101c9089b98f15826a"} Jul 14 15:33:05.268568 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:05.268141 2579 scope.go:117] "RemoveContainer" containerID="7766650c407b0005af923ee2d3747a9c411c1e66ddb9757757dc00d738b25f3e" Jul 14 15:33:05.281154 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:05.281135 2579 scope.go:117] "RemoveContainer" containerID="7766650c407b0005af923ee2d3747a9c411c1e66ddb9757757dc00d738b25f3e" Jul 14 15:33:05.281572 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:33:05.281548 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7766650c407b0005af923ee2d3747a9c411c1e66ddb9757757dc00d738b25f3e\": container with ID starting with 7766650c407b0005af923ee2d3747a9c411c1e66ddb9757757dc00d738b25f3e not found: ID does not exist" containerID="7766650c407b0005af923ee2d3747a9c411c1e66ddb9757757dc00d738b25f3e" Jul 14 15:33:05.281685 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:05.281579 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7766650c407b0005af923ee2d3747a9c411c1e66ddb9757757dc00d738b25f3e"} err="failed to get container status \"7766650c407b0005af923ee2d3747a9c411c1e66ddb9757757dc00d738b25f3e\": rpc error: code = NotFound desc = could not find container \"7766650c407b0005af923ee2d3747a9c411c1e66ddb9757757dc00d738b25f3e\": container with ID starting with 7766650c407b0005af923ee2d3747a9c411c1e66ddb9757757dc00d738b25f3e not found: ID does not exist" Jul 14 15:33:05.293967 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:05.293937 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57445f78c8-js2hn"] Jul 14 15:33:05.299991 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:05.299955 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57445f78c8-js2hn"] Jul 14 15:33:05.696551 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:05.696510 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95b4264d-ae58-4f6a-9046-ad9eecf8db86" path="/var/lib/kubelet/pods/95b4264d-ae58-4f6a-9046-ad9eecf8db86/volumes" Jul 14 15:33:38.276119 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:38.276082 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:33:47.973919 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:47.973880 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:33:59.776434 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:33:59.776392 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:34:06.916825 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:34:06.916779 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:34:08.397103 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:34:08.397062 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:34:19.567675 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:34:19.567638 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:34:21.540780 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:34:21.540689 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:34:22.895027 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:34:22.894995 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:34:59.286887 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:34:59.286848 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:34:59.387687 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:34:59.387648 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:35:05.150935 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:35:05.150900 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:35:05.208110 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:35:05.208071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:35:05.305742 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:35:05.305705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:35:05.405815 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:35:05.405724 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:35:05.551138 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:35:05.551103 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:35:11.707004 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:35:11.706970 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:35:11.771163 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:35:11.771123 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:35:28.584467 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:35:28.584424 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:35:38.472723 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:35:38.472687 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:35:46.876794 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:35:46.876756 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:35:56.474221 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:35:56.474124 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:35:59.648978 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:35:59.648948 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 15:35:59.655151 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:35:59.655123 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 15:36:06.782616 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:36:06.782570 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:36:16.276242 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:36:16.276202 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:36:27.978537 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:36:27.978499 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:36:37.977145 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:36:37.977103 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:37:01.577261 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:37:01.577226 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:37:04.530748 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:37:04.530709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:37:13.239366 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:37:13.239334 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:37:13.271936 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:37:13.271904 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:37:36.275063 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:37:36.275023 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:37:48.375038 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:37:48.375003 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:38:03.280454 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:38:03.280412 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:38:12.676363 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:38:12.676324 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:38:22.085654 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:38:22.085605 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:38:33.180844 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:38:33.180802 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:38:42.371962 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:38:42.371925 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:39:38.317526 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:39:38.317490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:39:39.371122 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:39:39.371087 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:40:32.494835 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:40:32.494796 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:40:42.104284 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:40:42.104245 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:40:55.010712 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:40:55.010669 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:40:59.683928 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:40:59.683898 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 15:40:59.686409 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:40:59.686384 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 15:41:07.207028 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:41:07.206986 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:41:18.202238 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:41:18.202197 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:41:28.094483 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:41:28.094437 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:41:38.104712 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:41:38.104676 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:41:48.297295 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:41:48.297261 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:41:59.300697 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:41:59.300617 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:42:08.593082 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:42:08.593045 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:42:21.496361 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:42:21.496330 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:42:44.294595 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:42:44.294562 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:42:53.397886 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:42:53.397846 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:43:03.594023 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:43:03.593986 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:43:13.402413 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:43:13.402370 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:43:20.498073 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:43:20.498039 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:43:27.201454 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:43:27.201387 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:43:45.897524 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:43:45.897485 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:43:56.100827 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:43:56.100786 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:44:05.697914 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:44:05.697881 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:44:12.650905 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:44:12.650864 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:44:17.404482 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:44:17.404436 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:44:28.295910 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:44:28.295869 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:44:46.896487 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:44:46.896441 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:45:04.300213 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:45:04.300093 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:45:13.208440 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:45:13.208403 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:45:22.014878 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:45:22.014841 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:45:22.074228 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:45:22.074157 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:45:25.401837 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:45:25.401807 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:45:42.699750 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:45:42.699699 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:45:55.200398 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:45:55.200351 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:45:59.724873 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:45:59.724843 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 15:45:59.726810 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:45:59.726774 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 15:46:06.592287 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:46:06.592251 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:46:14.176568 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:46:14.176525 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:46:14.273521 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:46:14.273465 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:46:19.085698 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:46:19.085653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:46:20.140885 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:46:20.140851 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:46:20.248328 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:46:20.248277 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:46:20.413294 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:46:20.413214 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:46:20.518273 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:46:20.518240 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:46:20.631052 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:46:20.631012 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:46:20.888187 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:46:20.888147 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:46:21.240516 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:46:21.240420 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:46:21.339926 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:46:21.339879 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:46:22.793232 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:46:22.793187 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:46:23.303064 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:46:23.303026 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:46:39.899631 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:46:39.899598 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:47:26.325382 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:47:26.325337 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:47:29.201255 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:47:29.201211 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:47:30.728614 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:47:30.728552 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:47:36.739352 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:47:36.739302 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:47:36.794267 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:47:36.794228 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:47:40.976223 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:47:40.976186 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:47:41.364512 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:47:41.364456 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:47:41.466682 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:47:41.466648 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:47:41.600534 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:47:41.600505 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:47:43.009897 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:47:43.009845 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:47:43.149257 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:47:43.149211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:47:44.400761 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:47:44.400705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:47:44.503558 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:47:44.503519 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:47:47.306766 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:47:47.306727 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:47:47.376678 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:47:47.376642 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:47:50.727614 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:47:50.727578 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:47:50.901115 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:47:50.901083 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:48:47.665418 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:48:47.665374 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:48:49.674842 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:48:49.674803 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:48:50.511761 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:48:50.511724 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:48:53.577813 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:48:53.577773 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:48:53.743814 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:48:53.743778 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:48:53.964348 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:48:53.964318 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:49:00.634782 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:49:00.634740 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:49:00.688407 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:49:00.688374 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:49:02.080269 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:49:02.080227 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:49:02.180760 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:49:02.180725 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:49:03.036594 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:49:03.036554 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:49:04.997796 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:49:04.997756 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:49:07.572789 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:49:07.572738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:49:07.672934 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:49:07.672892 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:49:46.618688 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:49:46.618642 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:49:46.719273 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:49:46.719200 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:49:57.002572 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:49:57.002531 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:49:57.102728 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:49:57.102694 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:50:07.000078 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:50:07.000041 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:50:09.996017 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:50:09.995984 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:50:11.507829 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:50:11.507790 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:50:17.617764 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:50:17.617730 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:50:17.719041 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:50:17.719005 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:50:22.647269 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:50:22.647223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:50:22.978468 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:50:22.978432 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:50:24.189426 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:50:24.189392 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:50:25.038084 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:50:25.038041 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:50:25.124939 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:50:25.124902 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:50:31.914055 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:50:31.914020 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:50:32.147101 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:50:32.147065 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:50:35.320296 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:50:35.320260 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:50:36.230574 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:50:36.230535 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:50:38.834181 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:50:38.834128 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:50:59.759191 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:50:59.759137 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 15:50:59.762463 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:50:59.762442 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 15:51:39.140531 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:51:39.140492 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:51:42.011352 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:51:42.011316 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:51:44.473202 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:51:44.473146 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:51:50.111729 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:51:50.111690 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:51:50.210322 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:51:50.210269 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:51:53.942309 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:51:53.942272 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:51:54.841568 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:51:54.841537 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:51:54.941129 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:51:54.941096 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:51:56.567843 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:51:56.567805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:52:00.262485 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:52:00.262437 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:52:03.425380 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:52:03.425344 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:52:03.527823 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:52:03.527793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:52:03.626165 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:52:03.626124 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:52:03.725153 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:52:03.725060 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:52:04.267203 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:52:04.267132 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:52:30.407232 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:52:30.407164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:52:36.655753 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:52:36.655713 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:52:36.756079 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:52:36.756040 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:52:36.857272 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:52:36.857238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:52:53.000426 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:52:53.000391 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:52:59.355850 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:52:59.355811 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:52:59.480361 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:52:59.480324 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:52:59.551519 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:52:59.551479 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:04.837779 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:04.837744 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:10.186878 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:10.186837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:11.093480 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:11.093438 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:17.817797 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:17.817762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:17.916044 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:17.916010 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:18.490198 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:18.490142 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:18.588902 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:18.588864 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:18.691610 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:18.691565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:20.196981 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:20.196945 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:53:20.564053 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:20.563962 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:26.520726 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:26.520685 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:26.627038 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:26.627001 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:27.285337 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:27.285292 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:29.195194 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:29.195127 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:53:33.432902 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:33.432861 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:33.494162 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:33.494126 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:33.708534 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:33.708443 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:34.109397 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:34.109360 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:34.522155 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:34.522068 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:37.563394 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:37.563360 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:37.645413 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:37.645373 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:37.716272 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:37.716215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:38.099073 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:38.099036 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:38.335073 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:38.335036 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:38.434345 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:38.434250 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:42.159088 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:42.159056 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:43.094966 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:43.094928 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:53:43.466779 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:43.466678 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:43.528351 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:43.528317 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:49.671793 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:49.671758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:53:51.588796 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:53:51.588712 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:00.635669 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:00.635626 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:03.404334 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:03.404294 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:54:05.205824 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:05.205784 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:11.312369 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:11.312334 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:11.388743 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:11.388707 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:15.973845 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:15.973806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:16.250046 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:16.249958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:16.365311 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:16.365276 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:16.461949 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:16.461915 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:16.619376 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:16.619327 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:22.565490 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:22.565445 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:22.672614 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:22.672577 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:22.766987 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:22.766950 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:22.989384 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:22.989347 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:29.028110 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:29.028071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:29.128118 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:29.128072 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:36.060358 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:36.060319 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:36.390010 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:36.389971 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:36.769651 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:36.769564 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:39.637368 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:39.637320 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:39.729946 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:39.729903 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:39.842349 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:39.842311 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:39.966478 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:39.966390 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:40.342144 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:40.342112 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:40.442320 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:40.442281 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:44.172843 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:44.172806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:46.607746 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:46.607705 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:54:47.644949 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:47.644912 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:54:47.743876 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:54:47.743842 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:55:01.215799 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:01.215759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:55:01.325238 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:01.325201 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:55:05.777837 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.777801 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg"] Jul 14 15:55:05.778349 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.778332 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95b4264d-ae58-4f6a-9046-ad9eecf8db86" containerName="authorino" Jul 14 15:55:05.778399 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.778353 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="95b4264d-ae58-4f6a-9046-ad9eecf8db86" containerName="authorino" Jul 14 15:55:05.778475 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.778465 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="95b4264d-ae58-4f6a-9046-ad9eecf8db86" containerName="authorino" Jul 14 15:55:05.781644 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.781624 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:05.784731 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.784706 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5e35a406\"/\"openshift-service-ca.crt\"" Jul 14 15:55:05.786516 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.786496 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5e35a406\"/\"kube-root-ca.crt\"" Jul 14 15:55:05.786516 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.786497 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5e35a406\"/\"e2e-lifecycle-model-5e35a406-kserve-self-signed-certs\"" Jul 14 15:55:05.786661 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.786527 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5e35a406\"/\"default-dockercfg-rlgw2\"" Jul 14 15:55:05.791299 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.791278 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg"] Jul 14 15:55:05.808734 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.808703 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:55:05.897670 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.897632 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e31782-5898-4ce7-b943-47363167e56b-tls-certs\") pod \"e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:05.897670 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.897674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-model-cache\") pod \"e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:05.897878 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.897697 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-tmp-dir\") pod \"e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:05.897878 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.897758 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5tm5\" (UniqueName: \"kubernetes.io/projected/00e31782-5898-4ce7-b943-47363167e56b-kube-api-access-c5tm5\") pod \"e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:05.897878 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.897810 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-home\") pod \"e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:05.898030 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.897900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-dshm\") pod \"e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:05.999070 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.999032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-dshm\") pod \"e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:05.999284 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.999089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e31782-5898-4ce7-b943-47363167e56b-tls-certs\") pod \"e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:05.999284 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.999129 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-model-cache\") pod \"e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:05.999284 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.999157 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-tmp-dir\") pod \"e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:05.999284 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.999209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5tm5\" (UniqueName: \"kubernetes.io/projected/00e31782-5898-4ce7-b943-47363167e56b-kube-api-access-c5tm5\") pod \"e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:05.999284 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.999243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-home\") pod \"e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:05.999572 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.999543 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-model-cache\") pod \"e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:05.999668 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.999594 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-tmp-dir\") pod \"e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:05.999668 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:05.999630 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-home\") pod \"e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:06.001585 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:06.001566 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-dshm\") pod \"e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:06.001760 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:06.001743 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e31782-5898-4ce7-b943-47363167e56b-tls-certs\") pod \"e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:06.008872 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:06.008839 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5tm5\" (UniqueName: \"kubernetes.io/projected/00e31782-5898-4ce7-b943-47363167e56b-kube-api-access-c5tm5\") pod \"e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:06.091637 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:06.091606 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:06.226207 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:06.226163 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg"] Jul 14 15:55:06.227602 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:55:06.227574 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00e31782_5898_4ce7_b943_47363167e56b.slice/crio-1c1687d34eae782581552aff87e91e79618836e6f654a4d55bcb38ecbc132452 WatchSource:0}: Error finding container 1c1687d34eae782581552aff87e91e79618836e6f654a4d55bcb38ecbc132452: Status 404 returned error can't find the container with id 1c1687d34eae782581552aff87e91e79618836e6f654a4d55bcb38ecbc132452 Jul 14 15:55:06.229617 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:06.229601 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 15:55:06.998619 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:06.998583 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" event={"ID":"00e31782-5898-4ce7-b943-47363167e56b","Type":"ContainerStarted","Data":"1c1687d34eae782581552aff87e91e79618836e6f654a4d55bcb38ecbc132452"} Jul 14 15:55:08.011512 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:08.011469 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" event={"ID":"00e31782-5898-4ce7-b943-47363167e56b","Type":"ContainerStarted","Data":"2e80b45356533016eb0129273695b65e1a260d348d7d8d845e1359b949d31f3d"} Jul 14 15:55:08.035010 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:08.034951 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" podStartSLOduration=1.881324947 podStartE2EDuration="3.034935392s" totalImagesPullingTime="1.153610445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:55:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:55:08.033609492 +0000 UTC m=+2048.885992658" watchObservedRunningTime="2026-07-14 15:55:08.034935392 +0000 UTC m=+2048.887318556" Jul 14 15:55:09.095558 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:09.095519 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:55:16.092642 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:16.092603 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:16.093129 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:16.092660 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:16.105343 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:16.105312 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:17.069565 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:17.069532 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:55:32.400810 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:32.400769 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:55:48.643513 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:48.643473 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:55:51.394690 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:51.394649 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:55:53.180900 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:53.180864 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:55:58.260823 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:58.260789 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg"] Jul 14 15:55:58.303031 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:58.303002 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:55:59.791621 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:59.791594 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 15:55:59.796873 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:55:59.796853 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 15:56:13.281629 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:13.281580 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" podUID="00e31782-5898-4ce7-b943-47363167e56b" containerName="main" containerID="cri-o://2e80b45356533016eb0129273695b65e1a260d348d7d8d845e1359b949d31f3d" gracePeriod=45 Jul 14 15:56:13.526033 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:13.526008 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:56:13.564766 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:13.564695 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-tmp-dir\") pod \"00e31782-5898-4ce7-b943-47363167e56b\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " Jul 14 15:56:13.564766 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:13.564748 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-dshm\") pod \"00e31782-5898-4ce7-b943-47363167e56b\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " Jul 14 15:56:13.564948 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:13.564813 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e31782-5898-4ce7-b943-47363167e56b-tls-certs\") pod \"00e31782-5898-4ce7-b943-47363167e56b\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " Jul 14 15:56:13.564948 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:13.564849 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5tm5\" (UniqueName: \"kubernetes.io/projected/00e31782-5898-4ce7-b943-47363167e56b-kube-api-access-c5tm5\") pod \"00e31782-5898-4ce7-b943-47363167e56b\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " Jul 14 15:56:13.564948 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:13.564888 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-home\") pod \"00e31782-5898-4ce7-b943-47363167e56b\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " Jul 14 15:56:13.564948 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:13.564941 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-model-cache\") pod \"00e31782-5898-4ce7-b943-47363167e56b\" (UID: \"00e31782-5898-4ce7-b943-47363167e56b\") " Jul 14 15:56:13.565165 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:13.564959 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "00e31782-5898-4ce7-b943-47363167e56b" (UID: "00e31782-5898-4ce7-b943-47363167e56b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:56:13.565165 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:13.565142 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-home" (OuterVolumeSpecName: "home") pod "00e31782-5898-4ce7-b943-47363167e56b" (UID: "00e31782-5898-4ce7-b943-47363167e56b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:56:13.565297 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:13.565222 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-tmp-dir\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:56:13.565439 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:13.565412 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-model-cache" (OuterVolumeSpecName: "model-cache") pod "00e31782-5898-4ce7-b943-47363167e56b" (UID: "00e31782-5898-4ce7-b943-47363167e56b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:56:13.567406 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:13.567332 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e31782-5898-4ce7-b943-47363167e56b-kube-api-access-c5tm5" (OuterVolumeSpecName: "kube-api-access-c5tm5") pod "00e31782-5898-4ce7-b943-47363167e56b" (UID: "00e31782-5898-4ce7-b943-47363167e56b"). InnerVolumeSpecName "kube-api-access-c5tm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:56:13.567535 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:13.567414 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-dshm" (OuterVolumeSpecName: "dshm") pod "00e31782-5898-4ce7-b943-47363167e56b" (UID: "00e31782-5898-4ce7-b943-47363167e56b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:56:13.567535 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:13.567428 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00e31782-5898-4ce7-b943-47363167e56b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00e31782-5898-4ce7-b943-47363167e56b" (UID: "00e31782-5898-4ce7-b943-47363167e56b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 15:56:13.666305 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:13.666276 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-dshm\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:56:13.666305 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:13.666301 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e31782-5898-4ce7-b943-47363167e56b-tls-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:56:13.666305 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:13.666312 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5tm5\" (UniqueName: \"kubernetes.io/projected/00e31782-5898-4ce7-b943-47363167e56b-kube-api-access-c5tm5\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:56:13.666514 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:13.666320 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-home\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:56:13.666514 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:13.666329 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e31782-5898-4ce7-b943-47363167e56b-model-cache\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:56:14.341681 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:14.341645 2579 generic.go:356] "Generic (PLEG): container finished" podID="00e31782-5898-4ce7-b943-47363167e56b" containerID="2e80b45356533016eb0129273695b65e1a260d348d7d8d845e1359b949d31f3d" exitCode=0 Jul 14 15:56:14.342147 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:14.341692 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" Jul 14 15:56:14.342147 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:14.341699 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" event={"ID":"00e31782-5898-4ce7-b943-47363167e56b","Type":"ContainerDied","Data":"2e80b45356533016eb0129273695b65e1a260d348d7d8d845e1359b949d31f3d"} Jul 14 15:56:14.342147 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:14.341792 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg" event={"ID":"00e31782-5898-4ce7-b943-47363167e56b","Type":"ContainerDied","Data":"1c1687d34eae782581552aff87e91e79618836e6f654a4d55bcb38ecbc132452"} Jul 14 15:56:14.342147 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:14.341809 2579 scope.go:117] "RemoveContainer" containerID="2e80b45356533016eb0129273695b65e1a260d348d7d8d845e1359b949d31f3d" Jul 14 15:56:14.351493 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:14.351472 2579 scope.go:117] "RemoveContainer" containerID="2e80b45356533016eb0129273695b65e1a260d348d7d8d845e1359b949d31f3d" Jul 14 15:56:14.351776 ip-10-0-128-22 kubenswrapper[2579]: E0714 15:56:14.351758 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e80b45356533016eb0129273695b65e1a260d348d7d8d845e1359b949d31f3d\": container with ID starting with 2e80b45356533016eb0129273695b65e1a260d348d7d8d845e1359b949d31f3d not found: ID does not exist" containerID="2e80b45356533016eb0129273695b65e1a260d348d7d8d845e1359b949d31f3d" Jul 14 15:56:14.351838 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:14.351796 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e80b45356533016eb0129273695b65e1a260d348d7d8d845e1359b949d31f3d"} err="failed to get container status \"2e80b45356533016eb0129273695b65e1a260d348d7d8d845e1359b949d31f3d\": rpc error: code = NotFound desc = could not find container \"2e80b45356533016eb0129273695b65e1a260d348d7d8d845e1359b949d31f3d\": container with ID starting with 2e80b45356533016eb0129273695b65e1a260d348d7d8d845e1359b949d31f3d not found: ID does not exist" Jul 14 15:56:14.360421 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:14.360397 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg"] Jul 14 15:56:14.363893 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:14.363869 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5e35a406/e2e-lifecycle-model-5e35a406-kserve-7fd48f7795-vmbcg"] Jul 14 15:56:15.701795 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:15.701759 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00e31782-5898-4ce7-b943-47363167e56b" path="/var/lib/kubelet/pods/00e31782-5898-4ce7-b943-47363167e56b/volumes" Jul 14 15:56:34.484789 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:34.484753 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:56:35.621033 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:35.620988 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:56:36.079190 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:36.079146 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:56:41.913931 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:41.913890 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:56:41.980293 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:41.980261 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:56:45.517368 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:45.517329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:56:45.742244 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:45.742207 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:56:46.829324 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:46.829283 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:56:46.924319 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:46.924282 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:56:47.031285 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:47.031250 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:56:47.844732 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:47.844699 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:56:47.941071 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:47.941044 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:56:48.043387 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:48.043356 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:56:49.450377 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:49.450342 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:56:52.891488 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:52.891405 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:56:55.793897 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:55.793862 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:56:55.901692 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:55.901661 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:56:56.477264 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:56:56.477223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:02.478974 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:02.478940 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:02.541952 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:02.541920 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:02.651042 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:02.651007 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:02.734403 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:02.734312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:03.992934 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:03.992902 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:57:13.263559 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:13.263521 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:13.625140 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:13.625103 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:34.639884 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:34.639852 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:40.752866 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:40.752811 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:40.814777 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:40.814749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:41.117785 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:41.117751 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:41.443585 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:41.443497 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:41.877139 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:41.877103 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:44.686603 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:44.686568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:44.768556 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:44.768521 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:44.879534 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:44.879487 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:45.357728 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:45.357693 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:45.459382 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:45.459353 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:45.560334 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:45.560301 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:49.187510 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:49.187468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:49.516420 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:49.516319 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:49.846991 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:49.846959 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:49.974966 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:49.974924 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:52.675079 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:52.675045 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:52.772346 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:52.772315 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:52.824235 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:52.824202 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:57:52.871284 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:52.871237 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:53.213904 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:53.213871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:53.473206 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:53.473115 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:53.572977 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:53.572944 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:57:57.185405 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:57:57.185371 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:58:04.407658 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:04.407617 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:58:06.282354 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:06.282311 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:58:06.381308 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:06.381270 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:58:07.987852 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:07.987808 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv"] Jul 14 15:58:07.988396 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:07.988332 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e31782-5898-4ce7-b943-47363167e56b" containerName="main" Jul 14 15:58:07.988396 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:07.988351 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e31782-5898-4ce7-b943-47363167e56b" containerName="main" Jul 14 15:58:07.988510 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:07.988476 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="00e31782-5898-4ce7-b943-47363167e56b" containerName="main" Jul 14 15:58:07.991515 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:07.991492 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:07.996152 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:07.995938 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-eb3623\"/\"openshift-service-ca.crt\"" Jul 14 15:58:07.996152 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:07.995994 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-eb3623\"/\"default-dockercfg-wtbvd\"" Jul 14 15:58:07.996152 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:07.995948 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-eb3623\"/\"kube-root-ca.crt\"" Jul 14 15:58:07.996152 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:07.996152 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-eb3623\"/\"test-model-eb3623-kserve-self-signed-certs\"" Jul 14 15:58:07.999513 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:07.999480 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv"] Jul 14 15:58:08.072490 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.072456 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:58:08.102649 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.102615 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-model-cache\") pod \"test-model-eb3623-kserve-75cc7dbf9f-thmsv\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:08.102811 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.102663 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-tmp-dir\") pod \"test-model-eb3623-kserve-75cc7dbf9f-thmsv\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:08.102811 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.102723 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzsbx\" (UniqueName: \"kubernetes.io/projected/7261b65f-3c92-48c0-a14a-40d2aa48fb85-kube-api-access-xzsbx\") pod \"test-model-eb3623-kserve-75cc7dbf9f-thmsv\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:08.102811 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.102777 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-home\") pod \"test-model-eb3623-kserve-75cc7dbf9f-thmsv\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:08.102811 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.102799 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-dshm\") pod \"test-model-eb3623-kserve-75cc7dbf9f-thmsv\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:08.103001 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.102896 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7261b65f-3c92-48c0-a14a-40d2aa48fb85-tls-certs\") pod \"test-model-eb3623-kserve-75cc7dbf9f-thmsv\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:08.204374 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.204326 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7261b65f-3c92-48c0-a14a-40d2aa48fb85-tls-certs\") pod \"test-model-eb3623-kserve-75cc7dbf9f-thmsv\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:08.204374 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.204381 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-model-cache\") pod \"test-model-eb3623-kserve-75cc7dbf9f-thmsv\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:08.204652 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.204421 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-tmp-dir\") pod \"test-model-eb3623-kserve-75cc7dbf9f-thmsv\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:08.204652 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.204463 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzsbx\" (UniqueName: \"kubernetes.io/projected/7261b65f-3c92-48c0-a14a-40d2aa48fb85-kube-api-access-xzsbx\") pod \"test-model-eb3623-kserve-75cc7dbf9f-thmsv\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:08.204652 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.204530 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-home\") pod \"test-model-eb3623-kserve-75cc7dbf9f-thmsv\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:08.204652 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.204557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-dshm\") pod \"test-model-eb3623-kserve-75cc7dbf9f-thmsv\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:08.204881 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.204828 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-model-cache\") pod \"test-model-eb3623-kserve-75cc7dbf9f-thmsv\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:08.204940 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.204878 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-tmp-dir\") pod \"test-model-eb3623-kserve-75cc7dbf9f-thmsv\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:08.204990 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.204967 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-home\") pod \"test-model-eb3623-kserve-75cc7dbf9f-thmsv\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:08.206975 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.206952 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-dshm\") pod \"test-model-eb3623-kserve-75cc7dbf9f-thmsv\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:08.207127 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.207108 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7261b65f-3c92-48c0-a14a-40d2aa48fb85-tls-certs\") pod \"test-model-eb3623-kserve-75cc7dbf9f-thmsv\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:08.214146 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.214109 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzsbx\" (UniqueName: \"kubernetes.io/projected/7261b65f-3c92-48c0-a14a-40d2aa48fb85-kube-api-access-xzsbx\") pod \"test-model-eb3623-kserve-75cc7dbf9f-thmsv\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:08.302349 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.302253 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:08.445445 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.445413 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv"] Jul 14 15:58:08.446515 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:58:08.446488 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7261b65f_3c92_48c0_a14a_40d2aa48fb85.slice/crio-b63058cb14656734cb821a6a6bd2f93f34c40ceba6cc841ad4806e67889563d8 WatchSource:0}: Error finding container b63058cb14656734cb821a6a6bd2f93f34c40ceba6cc841ad4806e67889563d8: Status 404 returned error can't find the container with id b63058cb14656734cb821a6a6bd2f93f34c40ceba6cc841ad4806e67889563d8 Jul 14 15:58:08.918542 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.918495 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" event={"ID":"7261b65f-3c92-48c0-a14a-40d2aa48fb85","Type":"ContainerStarted","Data":"a68b73791ebd7062e451bbee7b8dddee47b1ed5eb6817867b50e9fcddea29e9c"} Jul 14 15:58:08.918726 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.918553 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" event={"ID":"7261b65f-3c92-48c0-a14a-40d2aa48fb85","Type":"ContainerStarted","Data":"b63058cb14656734cb821a6a6bd2f93f34c40ceba6cc841ad4806e67889563d8"} Jul 14 15:58:08.938003 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:08.937948 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" podStartSLOduration=1.937932561 podStartE2EDuration="1.937932561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:58:08.936311324 +0000 UTC m=+2229.788694486" watchObservedRunningTime="2026-07-14 15:58:08.937932561 +0000 UTC m=+2229.790315747" Jul 14 15:58:12.598340 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:12.598298 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:58:14.293258 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:14.293229 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:58:14.394147 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:14.394100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:58:18.303383 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:18.303336 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:18.303872 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:18.303413 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:18.316793 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:18.316758 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:18.984113 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:18.984076 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:58:21.409457 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:21.409374 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:58:23.999473 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:23.999440 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:58:38.084206 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.084158 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc"] Jul 14 15:58:38.087470 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.087454 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:38.091879 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.091848 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-81addd\"/\"kube-root-ca.crt\"" Jul 14 15:58:38.091879 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.091866 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-81addd\"/\"default-dockercfg-5qzff\"" Jul 14 15:58:38.091879 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.091879 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-81addd\"/\"openshift-service-ca.crt\"" Jul 14 15:58:38.092154 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.091854 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-81addd\"/\"test-model-81addd-kserve-self-signed-certs\"" Jul 14 15:58:38.099838 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.099819 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc"] Jul 14 15:58:38.158835 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.158804 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b954ccee-73d2-4544-8667-a4f2cba68940-tls-certs\") pod \"test-model-81addd-kserve-599f685597-fqdsc\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:38.158835 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.158845 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjvgm\" (UniqueName: \"kubernetes.io/projected/b954ccee-73d2-4544-8667-a4f2cba68940-kube-api-access-zjvgm\") pod \"test-model-81addd-kserve-599f685597-fqdsc\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:38.159113 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.158880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-tmp-dir\") pod \"test-model-81addd-kserve-599f685597-fqdsc\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:38.159113 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.158940 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-home\") pod \"test-model-81addd-kserve-599f685597-fqdsc\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:38.159113 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.158969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-dshm\") pod \"test-model-81addd-kserve-599f685597-fqdsc\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:38.159113 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.158999 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-model-cache\") pod \"test-model-81addd-kserve-599f685597-fqdsc\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:38.186251 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.186219 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:58:38.260118 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.260079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-home\") pod \"test-model-81addd-kserve-599f685597-fqdsc\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:38.260298 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.260127 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-dshm\") pod \"test-model-81addd-kserve-599f685597-fqdsc\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:38.260298 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.260160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-model-cache\") pod \"test-model-81addd-kserve-599f685597-fqdsc\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:38.260298 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.260226 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b954ccee-73d2-4544-8667-a4f2cba68940-tls-certs\") pod \"test-model-81addd-kserve-599f685597-fqdsc\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:38.260470 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.260286 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjvgm\" (UniqueName: \"kubernetes.io/projected/b954ccee-73d2-4544-8667-a4f2cba68940-kube-api-access-zjvgm\") pod \"test-model-81addd-kserve-599f685597-fqdsc\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:38.260470 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.260357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-tmp-dir\") pod \"test-model-81addd-kserve-599f685597-fqdsc\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:38.260576 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.260513 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-home\") pod \"test-model-81addd-kserve-599f685597-fqdsc\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:38.260664 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.260644 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-model-cache\") pod \"test-model-81addd-kserve-599f685597-fqdsc\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:38.260745 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.260717 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-tmp-dir\") pod \"test-model-81addd-kserve-599f685597-fqdsc\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:38.262716 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.262695 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-dshm\") pod \"test-model-81addd-kserve-599f685597-fqdsc\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:38.262873 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.262854 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b954ccee-73d2-4544-8667-a4f2cba68940-tls-certs\") pod \"test-model-81addd-kserve-599f685597-fqdsc\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:38.272802 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.272782 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjvgm\" (UniqueName: \"kubernetes.io/projected/b954ccee-73d2-4544-8667-a4f2cba68940-kube-api-access-zjvgm\") pod \"test-model-81addd-kserve-599f685597-fqdsc\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:38.397888 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.397795 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:38.544879 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:38.544838 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc"] Jul 14 15:58:38.547921 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:58:38.547887 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb954ccee_73d2_4544_8667_a4f2cba68940.slice/crio-c97ece309282685cb849e5428d3af38dfbb76d592ded5a68008ed5cd2b2ed5a1 WatchSource:0}: Error finding container c97ece309282685cb849e5428d3af38dfbb76d592ded5a68008ed5cd2b2ed5a1: Status 404 returned error can't find the container with id c97ece309282685cb849e5428d3af38dfbb76d592ded5a68008ed5cd2b2ed5a1 Jul 14 15:58:39.080652 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:39.080595 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" event={"ID":"b954ccee-73d2-4544-8667-a4f2cba68940","Type":"ContainerStarted","Data":"ac3fc341acfb2d00d3cd52cc0b4e75aba501bc15aec3398c1ac5abad32f3529f"} Jul 14 15:58:39.080652 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:39.080652 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" event={"ID":"b954ccee-73d2-4544-8667-a4f2cba68940","Type":"ContainerStarted","Data":"c97ece309282685cb849e5428d3af38dfbb76d592ded5a68008ed5cd2b2ed5a1"} Jul 14 15:58:39.101608 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:39.101557 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" podStartSLOduration=1.101539116 podStartE2EDuration="1.101539116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:58:39.099117046 +0000 UTC m=+2259.951500204" watchObservedRunningTime="2026-07-14 15:58:39.101539116 +0000 UTC m=+2259.953922280" Jul 14 15:58:42.598459 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:42.598422 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:58:48.398395 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:48.398343 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:48.398876 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:48.398411 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:48.411134 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:48.411103 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:58:49.150070 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:58:49.150040 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:59:01.498062 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:01.498022 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:59:06.166624 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:06.166585 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv"] Jul 14 15:59:06.180615 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:06.180576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:59:07.071012 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:07.070974 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc"] Jul 14 15:59:07.328888 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:07.328796 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:59:07.857818 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:07.857781 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:59:07.958340 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:07.958308 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:59:17.700578 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:17.700524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:59:17.898206 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:17.898154 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:59:21.188315 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.188271 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" podUID="7261b65f-3c92-48c0-a14a-40d2aa48fb85" containerName="main" containerID="cri-o://a68b73791ebd7062e451bbee7b8dddee47b1ed5eb6817867b50e9fcddea29e9c" gracePeriod=45 Jul 14 15:59:21.305867 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.305793 2579 generic.go:356] "Generic (PLEG): container finished" podID="7261b65f-3c92-48c0-a14a-40d2aa48fb85" containerID="a68b73791ebd7062e451bbee7b8dddee47b1ed5eb6817867b50e9fcddea29e9c" exitCode=0 Jul 14 15:59:21.305867 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.305858 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" event={"ID":"7261b65f-3c92-48c0-a14a-40d2aa48fb85","Type":"ContainerDied","Data":"a68b73791ebd7062e451bbee7b8dddee47b1ed5eb6817867b50e9fcddea29e9c"} Jul 14 15:59:21.437188 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.437146 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:59:21.537758 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.537645 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-tmp-dir\") pod \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " Jul 14 15:59:21.537758 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.537765 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-dshm\") pod \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " Jul 14 15:59:21.538006 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.537789 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzsbx\" (UniqueName: \"kubernetes.io/projected/7261b65f-3c92-48c0-a14a-40d2aa48fb85-kube-api-access-xzsbx\") pod \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " Jul 14 15:59:21.538006 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.537813 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7261b65f-3c92-48c0-a14a-40d2aa48fb85-tls-certs\") pod \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " Jul 14 15:59:21.538006 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.537831 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-home\") pod \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " Jul 14 15:59:21.538006 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.537849 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-model-cache\") pod \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\" (UID: \"7261b65f-3c92-48c0-a14a-40d2aa48fb85\") " Jul 14 15:59:21.538006 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.537974 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7261b65f-3c92-48c0-a14a-40d2aa48fb85" (UID: "7261b65f-3c92-48c0-a14a-40d2aa48fb85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:59:21.538308 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.538106 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-tmp-dir\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:59:21.538308 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.538144 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-home" (OuterVolumeSpecName: "home") pod "7261b65f-3c92-48c0-a14a-40d2aa48fb85" (UID: "7261b65f-3c92-48c0-a14a-40d2aa48fb85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:59:21.538308 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.538204 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-model-cache" (OuterVolumeSpecName: "model-cache") pod "7261b65f-3c92-48c0-a14a-40d2aa48fb85" (UID: "7261b65f-3c92-48c0-a14a-40d2aa48fb85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:59:21.540269 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.540245 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7261b65f-3c92-48c0-a14a-40d2aa48fb85-kube-api-access-xzsbx" (OuterVolumeSpecName: "kube-api-access-xzsbx") pod "7261b65f-3c92-48c0-a14a-40d2aa48fb85" (UID: "7261b65f-3c92-48c0-a14a-40d2aa48fb85"). InnerVolumeSpecName "kube-api-access-xzsbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:59:21.540646 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.540620 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7261b65f-3c92-48c0-a14a-40d2aa48fb85-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7261b65f-3c92-48c0-a14a-40d2aa48fb85" (UID: "7261b65f-3c92-48c0-a14a-40d2aa48fb85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 15:59:21.540646 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.540627 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-dshm" (OuterVolumeSpecName: "dshm") pod "7261b65f-3c92-48c0-a14a-40d2aa48fb85" (UID: "7261b65f-3c92-48c0-a14a-40d2aa48fb85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:59:21.638692 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.638647 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-dshm\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:59:21.638692 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.638681 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzsbx\" (UniqueName: \"kubernetes.io/projected/7261b65f-3c92-48c0-a14a-40d2aa48fb85-kube-api-access-xzsbx\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:59:21.638912 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.638714 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7261b65f-3c92-48c0-a14a-40d2aa48fb85-tls-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:59:21.638912 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.638724 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-home\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:59:21.638912 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:21.638732 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7261b65f-3c92-48c0-a14a-40d2aa48fb85-model-cache\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:59:22.092448 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.092395 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" podUID="b954ccee-73d2-4544-8667-a4f2cba68940" containerName="main" containerID="cri-o://ac3fc341acfb2d00d3cd52cc0b4e75aba501bc15aec3398c1ac5abad32f3529f" gracePeriod=45 Jul 14 15:59:22.308969 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.308934 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:59:22.314876 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.314836 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" event={"ID":"7261b65f-3c92-48c0-a14a-40d2aa48fb85","Type":"ContainerDied","Data":"b63058cb14656734cb821a6a6bd2f93f34c40ceba6cc841ad4806e67889563d8"} Jul 14 15:59:22.315038 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.314891 2579 scope.go:117] "RemoveContainer" containerID="a68b73791ebd7062e451bbee7b8dddee47b1ed5eb6817867b50e9fcddea29e9c" Jul 14 15:59:22.315038 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.314845 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv" Jul 14 15:59:22.317006 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.316975 2579 generic.go:356] "Generic (PLEG): container finished" podID="b954ccee-73d2-4544-8667-a4f2cba68940" containerID="ac3fc341acfb2d00d3cd52cc0b4e75aba501bc15aec3398c1ac5abad32f3529f" exitCode=0 Jul 14 15:59:22.317165 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.317065 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" event={"ID":"b954ccee-73d2-4544-8667-a4f2cba68940","Type":"ContainerDied","Data":"ac3fc341acfb2d00d3cd52cc0b4e75aba501bc15aec3398c1ac5abad32f3529f"} Jul 14 15:59:22.334955 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.334921 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv"] Jul 14 15:59:22.342395 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.342360 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-eb3623/test-model-eb3623-kserve-75cc7dbf9f-thmsv"] Jul 14 15:59:22.365925 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.365893 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:59:22.445276 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.445236 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjvgm\" (UniqueName: \"kubernetes.io/projected/b954ccee-73d2-4544-8667-a4f2cba68940-kube-api-access-zjvgm\") pod \"b954ccee-73d2-4544-8667-a4f2cba68940\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " Jul 14 15:59:22.445450 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.445306 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-tmp-dir\") pod \"b954ccee-73d2-4544-8667-a4f2cba68940\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " Jul 14 15:59:22.445450 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.445338 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-home\") pod \"b954ccee-73d2-4544-8667-a4f2cba68940\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " Jul 14 15:59:22.445450 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.445360 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-model-cache\") pod \"b954ccee-73d2-4544-8667-a4f2cba68940\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " Jul 14 15:59:22.445450 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.445400 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-dshm\") pod \"b954ccee-73d2-4544-8667-a4f2cba68940\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " Jul 14 15:59:22.445450 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.445429 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b954ccee-73d2-4544-8667-a4f2cba68940-tls-certs\") pod \"b954ccee-73d2-4544-8667-a4f2cba68940\" (UID: \"b954ccee-73d2-4544-8667-a4f2cba68940\") " Jul 14 15:59:22.445721 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.445579 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-home" (OuterVolumeSpecName: "home") pod "b954ccee-73d2-4544-8667-a4f2cba68940" (UID: "b954ccee-73d2-4544-8667-a4f2cba68940"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:59:22.445721 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.445638 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b954ccee-73d2-4544-8667-a4f2cba68940" (UID: "b954ccee-73d2-4544-8667-a4f2cba68940"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:59:22.445721 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.445664 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-model-cache" (OuterVolumeSpecName: "model-cache") pod "b954ccee-73d2-4544-8667-a4f2cba68940" (UID: "b954ccee-73d2-4544-8667-a4f2cba68940"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:59:22.445721 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.445688 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-home\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:59:22.447819 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.447786 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b954ccee-73d2-4544-8667-a4f2cba68940-kube-api-access-zjvgm" (OuterVolumeSpecName: "kube-api-access-zjvgm") pod "b954ccee-73d2-4544-8667-a4f2cba68940" (UID: "b954ccee-73d2-4544-8667-a4f2cba68940"). InnerVolumeSpecName "kube-api-access-zjvgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:59:22.447819 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.447804 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b954ccee-73d2-4544-8667-a4f2cba68940-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b954ccee-73d2-4544-8667-a4f2cba68940" (UID: "b954ccee-73d2-4544-8667-a4f2cba68940"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 15:59:22.447957 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.447835 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-dshm" (OuterVolumeSpecName: "dshm") pod "b954ccee-73d2-4544-8667-a4f2cba68940" (UID: "b954ccee-73d2-4544-8667-a4f2cba68940"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:59:22.546689 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.546646 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-model-cache\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:59:22.546689 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.546681 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-dshm\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:59:22.546689 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.546692 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b954ccee-73d2-4544-8667-a4f2cba68940-tls-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:59:22.546950 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.546707 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjvgm\" (UniqueName: \"kubernetes.io/projected/b954ccee-73d2-4544-8667-a4f2cba68940-kube-api-access-zjvgm\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:59:22.546950 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:22.546720 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b954ccee-73d2-4544-8667-a4f2cba68940-tmp-dir\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:59:23.325597 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:23.325561 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" Jul 14 15:59:23.325597 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:23.325592 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc" event={"ID":"b954ccee-73d2-4544-8667-a4f2cba68940","Type":"ContainerDied","Data":"c97ece309282685cb849e5428d3af38dfbb76d592ded5a68008ed5cd2b2ed5a1"} Jul 14 15:59:23.326123 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:23.325637 2579 scope.go:117] "RemoveContainer" containerID="ac3fc341acfb2d00d3cd52cc0b4e75aba501bc15aec3398c1ac5abad32f3529f" Jul 14 15:59:23.349787 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:23.349751 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc"] Jul 14 15:59:23.354360 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:23.354324 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-81addd/test-model-81addd-kserve-599f685597-fqdsc"] Jul 14 15:59:23.698157 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:23.698059 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7261b65f-3c92-48c0-a14a-40d2aa48fb85" path="/var/lib/kubelet/pods/7261b65f-3c92-48c0-a14a-40d2aa48fb85/volumes" Jul 14 15:59:23.698497 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:23.698484 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b954ccee-73d2-4544-8667-a4f2cba68940" path="/var/lib/kubelet/pods/b954ccee-73d2-4544-8667-a4f2cba68940/volumes" Jul 14 15:59:28.607932 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:28.607897 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:59:28.643576 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:28.643528 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:59:28.705253 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:28.705212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:59:33.026993 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:33.026954 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:59:33.278276 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:33.278162 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:59:33.389703 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:33.389664 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-81addd-vd6z6"] Jul 14 15:59:33.390021 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:33.390009 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b954ccee-73d2-4544-8667-a4f2cba68940" containerName="main" Jul 14 15:59:33.390078 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:33.390023 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b954ccee-73d2-4544-8667-a4f2cba68940" containerName="main" Jul 14 15:59:33.390078 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:33.390035 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7261b65f-3c92-48c0-a14a-40d2aa48fb85" containerName="main" Jul 14 15:59:33.390078 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:33.390041 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7261b65f-3c92-48c0-a14a-40d2aa48fb85" containerName="main" Jul 14 15:59:33.390182 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:33.390110 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b954ccee-73d2-4544-8667-a4f2cba68940" containerName="main" Jul 14 15:59:33.390182 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:33.390119 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7261b65f-3c92-48c0-a14a-40d2aa48fb85" containerName="main" Jul 14 15:59:33.393923 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:33.393905 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-81addd-vd6z6" Jul 14 15:59:33.396967 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:33.396945 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4dcsv\"" Jul 14 15:59:33.397068 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:33.397034 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 15:59:33.401114 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:33.401090 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-81addd-vd6z6"] Jul 14 15:59:33.444295 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:33.444261 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/931a3f4e-484e-4751-8237-8544ef36df1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-81addd-vd6z6\" (UID: \"931a3f4e-484e-4751-8237-8544ef36df1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-81addd-vd6z6" Jul 14 15:59:33.545376 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:33.545288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/931a3f4e-484e-4751-8237-8544ef36df1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-81addd-vd6z6\" (UID: \"931a3f4e-484e-4751-8237-8544ef36df1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-81addd-vd6z6" Jul 14 15:59:33.545951 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:33.545931 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/931a3f4e-484e-4751-8237-8544ef36df1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-81addd-vd6z6\" (UID: \"931a3f4e-484e-4751-8237-8544ef36df1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-81addd-vd6z6" Jul 14 15:59:33.706425 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:33.706393 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-81addd-vd6z6" Jul 14 15:59:33.843635 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:33.843607 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-81addd-vd6z6"] Jul 14 15:59:33.845519 ip-10-0-128-22 kubenswrapper[2579]: W0714 15:59:33.845497 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931a3f4e_484e_4751_8237_8544ef36df1e.slice/crio-ff4e466ddb8b7f538591863ae816a72ff5989990855c8e28ca1aea58f9b3eacc WatchSource:0}: Error finding container ff4e466ddb8b7f538591863ae816a72ff5989990855c8e28ca1aea58f9b3eacc: Status 404 returned error can't find the container with id ff4e466ddb8b7f538591863ae816a72ff5989990855c8e28ca1aea58f9b3eacc Jul 14 15:59:33.935041 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:33.935006 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:59:34.041706 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:34.041673 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:59:34.391535 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:34.391496 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-81addd-vd6z6" event={"ID":"931a3f4e-484e-4751-8237-8544ef36df1e","Type":"ContainerStarted","Data":"ff4e466ddb8b7f538591863ae816a72ff5989990855c8e28ca1aea58f9b3eacc"} Jul 14 15:59:37.409601 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:37.409566 2579 generic.go:356] "Generic (PLEG): container finished" podID="931a3f4e-484e-4751-8237-8544ef36df1e" containerID="df416044201116139a8b682945ce99e2ef78c7b59fc8617f34a3c90c6e382e95" exitCode=0 Jul 14 15:59:37.409980 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:37.409614 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-81addd-vd6z6" event={"ID":"931a3f4e-484e-4751-8237-8544ef36df1e","Type":"ContainerDied","Data":"df416044201116139a8b682945ce99e2ef78c7b59fc8617f34a3c90c6e382e95"} Jul 14 15:59:38.545440 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:38.545413 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-81addd-vd6z6" Jul 14 15:59:38.589627 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:38.589597 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/931a3f4e-484e-4751-8237-8544ef36df1e-maas-api-service-ca\") pod \"931a3f4e-484e-4751-8237-8544ef36df1e\" (UID: \"931a3f4e-484e-4751-8237-8544ef36df1e\") " Jul 14 15:59:38.589916 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:38.589895 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931a3f4e-484e-4751-8237-8544ef36df1e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "931a3f4e-484e-4751-8237-8544ef36df1e" (UID: "931a3f4e-484e-4751-8237-8544ef36df1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 15:59:38.690721 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:38.690639 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/931a3f4e-484e-4751-8237-8544ef36df1e-maas-api-service-ca\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 15:59:39.421719 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:39.421691 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-81addd-vd6z6" Jul 14 15:59:39.421719 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:39.421703 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-81addd-vd6z6" event={"ID":"931a3f4e-484e-4751-8237-8544ef36df1e","Type":"ContainerDied","Data":"ff4e466ddb8b7f538591863ae816a72ff5989990855c8e28ca1aea58f9b3eacc"} Jul 14 15:59:39.421930 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:39.421745 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff4e466ddb8b7f538591863ae816a72ff5989990855c8e28ca1aea58f9b3eacc" Jul 14 15:59:43.905038 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:43.905007 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:59:47.300891 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:47.300856 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 15:59:48.486711 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:48.486649 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:59:48.586982 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:48.586948 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:59:48.688153 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:48.688120 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:59:54.427695 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:54.427600 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:59:54.487902 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:54.487861 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:59:59.210191 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:59.210144 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:59:59.449529 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:59.449486 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 15:59:59.862189 ip-10-0-128-22 kubenswrapper[2579]: I0714 15:59:59.862137 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:00.765278 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:00.765245 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:00.822548 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:00.822513 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:03.487382 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:03.487336 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:03.574932 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:03.574879 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:03.641424 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:03.641387 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:04.117833 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:04.117790 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:04.289095 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:04.289060 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:04.402287 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:04.402195 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:08.004155 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:08.004119 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:08.155143 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:08.155103 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:00:08.510400 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:08.510360 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:08.879486 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:08.879432 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:09.377103 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:09.377062 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:11.637341 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:11.637305 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:11.723505 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:11.723441 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:11.803898 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:11.803855 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:12.038157 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:12.038057 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:12.461313 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:12.461265 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:12.555434 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:12.555390 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:16.161714 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:16.161671 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:20.596296 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:20.596252 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:00:24.368241 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:24.368200 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-768d39-gpcbk"] Jul 14 16:00:24.368649 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:24.368551 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="931a3f4e-484e-4751-8237-8544ef36df1e" containerName="revoke-keys" Jul 14 16:00:24.368649 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:24.368562 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="931a3f4e-484e-4751-8237-8544ef36df1e" containerName="revoke-keys" Jul 14 16:00:24.368649 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:24.368620 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="931a3f4e-484e-4751-8237-8544ef36df1e" containerName="revoke-keys" Jul 14 16:00:24.371816 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:24.371798 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-768d39-gpcbk" Jul 14 16:00:24.374920 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:24.374897 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 16:00:24.375073 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:24.374921 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4dcsv\"" Jul 14 16:00:24.397648 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:24.397609 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-768d39-gpcbk"] Jul 14 16:00:24.498656 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:24.498618 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/163004ce-86d4-4d9f-8fbe-861c1fd4cfda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-768d39-gpcbk\" (UID: \"163004ce-86d4-4d9f-8fbe-861c1fd4cfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-768d39-gpcbk" Jul 14 16:00:24.599285 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:24.599245 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/163004ce-86d4-4d9f-8fbe-861c1fd4cfda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-768d39-gpcbk\" (UID: \"163004ce-86d4-4d9f-8fbe-861c1fd4cfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-768d39-gpcbk" Jul 14 16:00:24.599908 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:24.599883 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/163004ce-86d4-4d9f-8fbe-861c1fd4cfda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-768d39-gpcbk\" (UID: \"163004ce-86d4-4d9f-8fbe-861c1fd4cfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-768d39-gpcbk" Jul 14 16:00:24.682331 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:24.682235 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-768d39-gpcbk" Jul 14 16:00:24.828566 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:24.828536 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-768d39-gpcbk"] Jul 14 16:00:24.830767 ip-10-0-128-22 kubenswrapper[2579]: W0714 16:00:24.830734 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163004ce_86d4_4d9f_8fbe_861c1fd4cfda.slice/crio-9954830d432143da24afc29888a4b9c89e1444404f16fd2ff253a85fd1fa8ddf WatchSource:0}: Error finding container 9954830d432143da24afc29888a4b9c89e1444404f16fd2ff253a85fd1fa8ddf: Status 404 returned error can't find the container with id 9954830d432143da24afc29888a4b9c89e1444404f16fd2ff253a85fd1fa8ddf Jul 14 16:00:25.672109 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:25.672066 2579 generic.go:356] "Generic (PLEG): container finished" podID="163004ce-86d4-4d9f-8fbe-861c1fd4cfda" containerID="0a689959e85e832a0cccb73f07686070d955cc8eaeb598b506c1a79c96a90f57" exitCode=0 Jul 14 16:00:25.672550 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:25.672116 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-768d39-gpcbk" event={"ID":"163004ce-86d4-4d9f-8fbe-861c1fd4cfda","Type":"ContainerDied","Data":"0a689959e85e832a0cccb73f07686070d955cc8eaeb598b506c1a79c96a90f57"} Jul 14 16:00:25.672550 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:25.672199 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-768d39-gpcbk" event={"ID":"163004ce-86d4-4d9f-8fbe-861c1fd4cfda","Type":"ContainerStarted","Data":"9954830d432143da24afc29888a4b9c89e1444404f16fd2ff253a85fd1fa8ddf"} Jul 14 16:00:26.611315 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:26.611272 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:26.663773 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:26.663736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:26.811403 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:26.811375 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-768d39-gpcbk" Jul 14 16:00:26.918795 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:26.918703 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/163004ce-86d4-4d9f-8fbe-861c1fd4cfda-maas-api-service-ca\") pod \"163004ce-86d4-4d9f-8fbe-861c1fd4cfda\" (UID: \"163004ce-86d4-4d9f-8fbe-861c1fd4cfda\") " Jul 14 16:00:26.919104 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:26.919081 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/163004ce-86d4-4d9f-8fbe-861c1fd4cfda-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "163004ce-86d4-4d9f-8fbe-861c1fd4cfda" (UID: "163004ce-86d4-4d9f-8fbe-861c1fd4cfda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 16:00:27.019428 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:27.019397 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/163004ce-86d4-4d9f-8fbe-861c1fd4cfda-maas-api-service-ca\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:00:27.684711 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:27.684677 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-768d39-gpcbk" Jul 14 16:00:27.684711 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:27.684694 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-768d39-gpcbk" event={"ID":"163004ce-86d4-4d9f-8fbe-861c1fd4cfda","Type":"ContainerDied","Data":"9954830d432143da24afc29888a4b9c89e1444404f16fd2ff253a85fd1fa8ddf"} Jul 14 16:00:27.684934 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:27.684741 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9954830d432143da24afc29888a4b9c89e1444404f16fd2ff253a85fd1fa8ddf" Jul 14 16:00:33.307185 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:33.307131 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:33.408498 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:33.408461 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:34.991699 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:34.991653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:35.091527 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:35.091484 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:35.298618 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:35.298529 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:38.999120 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:38.999083 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:00:39.472577 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:39.472543 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:45.938538 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:45.938495 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:46.000622 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:46.000572 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:50.264919 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:50.264885 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:50.496884 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:50.496848 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:50.642303 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:50.642262 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e29de-nhv9h"] Jul 14 16:00:50.642622 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:50.642608 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163004ce-86d4-4d9f-8fbe-861c1fd4cfda" containerName="revoke-keys" Jul 14 16:00:50.642666 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:50.642625 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="163004ce-86d4-4d9f-8fbe-861c1fd4cfda" containerName="revoke-keys" Jul 14 16:00:50.642714 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:50.642705 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="163004ce-86d4-4d9f-8fbe-861c1fd4cfda" containerName="revoke-keys" Jul 14 16:00:50.645945 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:50.645926 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e29de-nhv9h" Jul 14 16:00:50.649021 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:50.648995 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4dcsv\"" Jul 14 16:00:50.649206 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:50.649005 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 16:00:50.654132 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:50.653710 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e29de-nhv9h"] Jul 14 16:00:50.733350 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:50.733306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6115b3-f640-4003-9754-80166826250c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0e29de-nhv9h\" (UID: \"6d6115b3-f640-4003-9754-80166826250c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e29de-nhv9h" Jul 14 16:00:50.834477 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:50.834437 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6115b3-f640-4003-9754-80166826250c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0e29de-nhv9h\" (UID: \"6d6115b3-f640-4003-9754-80166826250c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e29de-nhv9h" Jul 14 16:00:50.835304 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:50.835282 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6115b3-f640-4003-9754-80166826250c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0e29de-nhv9h\" (UID: \"6d6115b3-f640-4003-9754-80166826250c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e29de-nhv9h" Jul 14 16:00:50.959029 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:50.958935 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e29de-nhv9h" Jul 14 16:00:51.107746 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:51.107719 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e29de-nhv9h"] Jul 14 16:00:51.108648 ip-10-0-128-22 kubenswrapper[2579]: W0714 16:00:51.108623 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6115b3_f640_4003_9754_80166826250c.slice/crio-215fec58de81eb9b0f8b4a69ec21d11a26d0a90490468ae7b142dc9942110639 WatchSource:0}: Error finding container 215fec58de81eb9b0f8b4a69ec21d11a26d0a90490468ae7b142dc9942110639: Status 404 returned error can't find the container with id 215fec58de81eb9b0f8b4a69ec21d11a26d0a90490468ae7b142dc9942110639 Jul 14 16:00:51.350880 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:51.350840 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:51.451094 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:51.451051 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:00:51.812274 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:51.812237 2579 generic.go:356] "Generic (PLEG): container finished" podID="6d6115b3-f640-4003-9754-80166826250c" containerID="ad75e431bab714f4eabd7869efb595a60f8422b161a9274ad41b100762e0c1fa" exitCode=0 Jul 14 16:00:51.812482 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:51.812328 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e29de-nhv9h" event={"ID":"6d6115b3-f640-4003-9754-80166826250c","Type":"ContainerDied","Data":"ad75e431bab714f4eabd7869efb595a60f8422b161a9274ad41b100762e0c1fa"} Jul 14 16:00:51.812482 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:51.812394 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e29de-nhv9h" event={"ID":"6d6115b3-f640-4003-9754-80166826250c","Type":"ContainerStarted","Data":"215fec58de81eb9b0f8b4a69ec21d11a26d0a90490468ae7b142dc9942110639"} Jul 14 16:00:52.951668 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:52.951640 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e29de-nhv9h" Jul 14 16:00:53.058677 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:53.058624 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6115b3-f640-4003-9754-80166826250c-maas-api-service-ca\") pod \"6d6115b3-f640-4003-9754-80166826250c\" (UID: \"6d6115b3-f640-4003-9754-80166826250c\") " Jul 14 16:00:53.058983 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:53.058959 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d6115b3-f640-4003-9754-80166826250c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d6115b3-f640-4003-9754-80166826250c" (UID: "6d6115b3-f640-4003-9754-80166826250c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 16:00:53.159948 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:53.159901 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6115b3-f640-4003-9754-80166826250c-maas-api-service-ca\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:00:53.824753 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:53.824716 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e29de-nhv9h" event={"ID":"6d6115b3-f640-4003-9754-80166826250c","Type":"ContainerDied","Data":"215fec58de81eb9b0f8b4a69ec21d11a26d0a90490468ae7b142dc9942110639"} Jul 14 16:00:53.824967 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:53.824776 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="215fec58de81eb9b0f8b4a69ec21d11a26d0a90490468ae7b142dc9942110639" Jul 14 16:00:53.824967 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:53.824730 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e29de-nhv9h" Jul 14 16:00:59.826762 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:59.826730 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 16:00:59.834584 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:00:59.834558 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 16:01:01.202117 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:01.202070 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:04.102341 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:04.102301 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:01:05.731567 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:05.731527 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:11.857057 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:11.857017 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:11.956976 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:11.956942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:16.443435 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:16.443389 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:16.698225 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:16.698085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:17.076429 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:17.076380 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:17.412206 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:17.412085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:17.681068 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:17.680974 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:19.969484 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:19.969449 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:20.103160 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:20.103124 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:20.203832 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:20.203791 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:20.338681 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:20.338642 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:20.788351 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:20.788256 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:20.893993 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:20.893954 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:24.527395 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:24.527361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:24.866837 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:24.866797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:25.205312 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:25.205222 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:25.403841 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:25.403788 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:01:25.542452 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:25.542342 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:28.047541 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:28.047502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:28.134372 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:28.134333 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:30.431435 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:30.431394 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:30.538341 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:30.538304 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:32.567468 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:32.567408 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:38.200292 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:38.200245 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:01:42.806360 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:42.806318 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:42.908129 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:42.908096 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:49.297872 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.297825 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz"] Jul 14 16:01:49.298400 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.298323 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d6115b3-f640-4003-9754-80166826250c" containerName="revoke-keys" Jul 14 16:01:49.298400 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.298341 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d6115b3-f640-4003-9754-80166826250c" containerName="revoke-keys" Jul 14 16:01:49.298527 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.298411 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d6115b3-f640-4003-9754-80166826250c" containerName="revoke-keys" Jul 14 16:01:49.300986 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.300967 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:49.304159 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.304139 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bea800\"/\"openshift-service-ca.crt\"" Jul 14 16:01:49.304290 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.304159 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bea800\"/\"kube-root-ca.crt\"" Jul 14 16:01:49.305731 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.305715 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bea800\"/\"default-dockercfg-c7tgc\"" Jul 14 16:01:49.305835 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.305744 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bea800\"/\"auth-test-model-bea800-kserve-self-signed-certs\"" Jul 14 16:01:49.316458 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.316431 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz"] Jul 14 16:01:49.331424 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.331389 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-home\") pod \"auth-test-model-bea800-kserve-6ddb7566c9-kv7qz\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:49.331424 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.331422 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-model-cache\") pod \"auth-test-model-bea800-kserve-6ddb7566c9-kv7qz\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:49.331648 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.331444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cqzt\" (UniqueName: \"kubernetes.io/projected/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-kube-api-access-2cqzt\") pod \"auth-test-model-bea800-kserve-6ddb7566c9-kv7qz\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:49.331648 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.331553 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-tls-certs\") pod \"auth-test-model-bea800-kserve-6ddb7566c9-kv7qz\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:49.331648 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.331595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-tmp-dir\") pod \"auth-test-model-bea800-kserve-6ddb7566c9-kv7qz\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:49.331648 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.331626 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-dshm\") pod \"auth-test-model-bea800-kserve-6ddb7566c9-kv7qz\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:49.340718 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.340689 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:49.432741 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.432704 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-tls-certs\") pod \"auth-test-model-bea800-kserve-6ddb7566c9-kv7qz\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:49.432741 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.432745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-tmp-dir\") pod \"auth-test-model-bea800-kserve-6ddb7566c9-kv7qz\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:49.432956 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.432770 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-dshm\") pod \"auth-test-model-bea800-kserve-6ddb7566c9-kv7qz\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:49.432956 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.432879 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-home\") pod \"auth-test-model-bea800-kserve-6ddb7566c9-kv7qz\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:49.432956 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.432910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-model-cache\") pod \"auth-test-model-bea800-kserve-6ddb7566c9-kv7qz\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:49.432956 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.432939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cqzt\" (UniqueName: \"kubernetes.io/projected/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-kube-api-access-2cqzt\") pod \"auth-test-model-bea800-kserve-6ddb7566c9-kv7qz\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:49.433222 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.433165 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-tmp-dir\") pod \"auth-test-model-bea800-kserve-6ddb7566c9-kv7qz\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:49.433305 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.433240 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-home\") pod \"auth-test-model-bea800-kserve-6ddb7566c9-kv7qz\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:49.433305 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.433292 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-model-cache\") pod \"auth-test-model-bea800-kserve-6ddb7566c9-kv7qz\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:49.435324 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.435300 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-dshm\") pod \"auth-test-model-bea800-kserve-6ddb7566c9-kv7qz\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:49.435659 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.435639 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-tls-certs\") pod \"auth-test-model-bea800-kserve-6ddb7566c9-kv7qz\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:49.443006 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.442984 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cqzt\" (UniqueName: \"kubernetes.io/projected/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-kube-api-access-2cqzt\") pod \"auth-test-model-bea800-kserve-6ddb7566c9-kv7qz\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:49.611265 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.611225 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:49.673600 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.673557 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:49.755006 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.754977 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz"] Jul 14 16:01:49.756539 ip-10-0-128-22 kubenswrapper[2579]: W0714 16:01:49.756512 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a9faae4_ac8b_4840_bfc7_3729ef9cc56d.slice/crio-e28a3d30275c222150d1fe89d8b1cd49214a27cca138b57f8cdbfda8642168a9 WatchSource:0}: Error finding container e28a3d30275c222150d1fe89d8b1cd49214a27cca138b57f8cdbfda8642168a9: Status 404 returned error can't find the container with id e28a3d30275c222150d1fe89d8b1cd49214a27cca138b57f8cdbfda8642168a9 Jul 14 16:01:49.792043 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:49.792009 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:50.114268 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:50.114229 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" event={"ID":"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d","Type":"ContainerStarted","Data":"c2b9ec5e269426fe0a894f1367b1326d51f3e129ed3337144f018b4225bea701"} Jul 14 16:01:50.114446 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:50.114280 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" event={"ID":"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d","Type":"ContainerStarted","Data":"e28a3d30275c222150d1fe89d8b1cd49214a27cca138b57f8cdbfda8642168a9"} Jul 14 16:01:50.136189 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:50.136118 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" podStartSLOduration=1.13610287 podStartE2EDuration="1.13610287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 16:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 16:01:50.134061943 +0000 UTC m=+2450.986445108" watchObservedRunningTime="2026-07-14 16:01:50.13610287 +0000 UTC m=+2450.988486034" Jul 14 16:01:51.550287 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:51.550250 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:52.800710 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:52.800674 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:01:54.617302 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:54.617266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:55.905917 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:55.905883 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:01:59.611407 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:59.611378 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:59.611980 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:59.611429 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:01:59.624273 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:01:59.624245 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:02:00.174690 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:00.174661 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:02:48.209449 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.209414 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c"] Jul 14 16:02:48.212640 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.212621 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:48.215631 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.215604 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-519447\"/\"kube-root-ca.crt\"" Jul 14 16:02:48.217273 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.217252 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-519447\"/\"default-dockercfg-r57n8\"" Jul 14 16:02:48.217418 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.217291 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-519447\"/\"openshift-service-ca.crt\"" Jul 14 16:02:48.217418 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.217296 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-519447\"/\"auth-test-model-519447-kserve-self-signed-certs\"" Jul 14 16:02:48.224424 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.224404 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c"] Jul 14 16:02:48.228420 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.228395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:02:48.301517 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.301476 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-dshm\") pod \"auth-test-model-519447-kserve-5974c9979-chx7c\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:48.301517 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.301519 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-tmp-dir\") pod \"auth-test-model-519447-kserve-5974c9979-chx7c\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:48.301703 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.301536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjmtz\" (UniqueName: \"kubernetes.io/projected/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-kube-api-access-sjmtz\") pod \"auth-test-model-519447-kserve-5974c9979-chx7c\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:48.301703 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.301586 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-model-cache\") pod \"auth-test-model-519447-kserve-5974c9979-chx7c\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:48.301703 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.301657 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-home\") pod \"auth-test-model-519447-kserve-5974c9979-chx7c\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:48.301840 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.301720 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-tls-certs\") pod \"auth-test-model-519447-kserve-5974c9979-chx7c\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:48.402597 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.402563 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-model-cache\") pod \"auth-test-model-519447-kserve-5974c9979-chx7c\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:48.402715 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.402607 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-home\") pod \"auth-test-model-519447-kserve-5974c9979-chx7c\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:48.402715 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.402652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-tls-certs\") pod \"auth-test-model-519447-kserve-5974c9979-chx7c\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:48.402715 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.402676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-dshm\") pod \"auth-test-model-519447-kserve-5974c9979-chx7c\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:48.402715 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.402690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-tmp-dir\") pod \"auth-test-model-519447-kserve-5974c9979-chx7c\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:48.402715 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.402706 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjmtz\" (UniqueName: \"kubernetes.io/projected/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-kube-api-access-sjmtz\") pod \"auth-test-model-519447-kserve-5974c9979-chx7c\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:48.403029 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.403007 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-model-cache\") pod \"auth-test-model-519447-kserve-5974c9979-chx7c\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:48.403099 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.403065 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-home\") pod \"auth-test-model-519447-kserve-5974c9979-chx7c\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:48.403162 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.403132 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-tmp-dir\") pod \"auth-test-model-519447-kserve-5974c9979-chx7c\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:48.404955 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.404939 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-dshm\") pod \"auth-test-model-519447-kserve-5974c9979-chx7c\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:48.405301 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.405285 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-tls-certs\") pod \"auth-test-model-519447-kserve-5974c9979-chx7c\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:48.411596 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.411575 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjmtz\" (UniqueName: \"kubernetes.io/projected/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-kube-api-access-sjmtz\") pod \"auth-test-model-519447-kserve-5974c9979-chx7c\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:48.522394 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.522330 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:48.657549 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:48.657520 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c"] Jul 14 16:02:48.659722 ip-10-0-128-22 kubenswrapper[2579]: W0714 16:02:48.659691 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc60343b_bc6b_4a2a_8ea1_1be93bc44bfb.slice/crio-da5ef16068775dae4e8c3f99110f40e623d698046103dfacc21b9cb536db6a7b WatchSource:0}: Error finding container da5ef16068775dae4e8c3f99110f40e623d698046103dfacc21b9cb536db6a7b: Status 404 returned error can't find the container with id da5ef16068775dae4e8c3f99110f40e623d698046103dfacc21b9cb536db6a7b Jul 14 16:02:49.441709 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:49.441664 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" event={"ID":"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb","Type":"ContainerStarted","Data":"173f06ca7ed6cb51f59e17dd0d65ae4edd018b93b15d2067e59676c3b25d325d"} Jul 14 16:02:49.442147 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:49.441715 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" event={"ID":"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb","Type":"ContainerStarted","Data":"da5ef16068775dae4e8c3f99110f40e623d698046103dfacc21b9cb536db6a7b"} Jul 14 16:02:49.459412 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:49.459362 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" podStartSLOduration=1.459343824 podStartE2EDuration="1.459343824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 16:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 16:02:49.459336379 +0000 UTC m=+2510.311719546" watchObservedRunningTime="2026-07-14 16:02:49.459343824 +0000 UTC m=+2510.311726991" Jul 14 16:02:52.897356 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:52.897264 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:02:56.404469 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:56.404415 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:02:58.523323 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:58.523285 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:58.523795 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:58.523339 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:58.535665 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:58.535639 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:02:59.513389 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:02:59.513358 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:03:07.500680 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:03:07.500640 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:03:09.159627 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:03:09.159594 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:03:26.601627 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:03:26.601593 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:03:41.001743 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:03:41.001707 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:03:43.285337 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:03:43.285307 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:03:58.591033 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:03:58.590996 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:04:16.722336 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:04:16.722301 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:04:35.794933 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:04:35.794898 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:04:49.343344 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:04:49.343311 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:04:50.783078 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:04:50.783047 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:04:51.068149 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:04:51.068077 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz"] Jul 14 16:04:51.099106 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:04:51.099081 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:04:51.873586 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:04:51.873555 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c"] Jul 14 16:04:51.896716 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:04:51.896690 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:04:55.001682 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:04:55.001650 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:05:02.086187 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.086144 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx"] Jul 14 16:05:02.089405 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.089389 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:02.092459 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.092441 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bea800\"/\"sub-test-model-bea800-kserve-self-signed-certs\"" Jul 14 16:05:02.100311 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.100288 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx"] Jul 14 16:05:02.112998 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.112934 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:05:02.202925 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.202898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-tmp-dir\") pod \"sub-test-model-bea800-kserve-5b9646d9bc-gxjkx\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:02.202925 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.202932 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-model-cache\") pod \"sub-test-model-bea800-kserve-5b9646d9bc-gxjkx\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:02.203161 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.202969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-dshm\") pod \"sub-test-model-bea800-kserve-5b9646d9bc-gxjkx\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:02.203161 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.203046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-home\") pod \"sub-test-model-bea800-kserve-5b9646d9bc-gxjkx\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:02.203161 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.203091 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13cfc744-f084-44f3-8669-75f34a8eef08-tls-certs\") pod \"sub-test-model-bea800-kserve-5b9646d9bc-gxjkx\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:02.203161 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.203118 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj5vx\" (UniqueName: \"kubernetes.io/projected/13cfc744-f084-44f3-8669-75f34a8eef08-kube-api-access-dj5vx\") pod \"sub-test-model-bea800-kserve-5b9646d9bc-gxjkx\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:02.303620 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.303588 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-model-cache\") pod \"sub-test-model-bea800-kserve-5b9646d9bc-gxjkx\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:02.303770 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.303633 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-dshm\") pod \"sub-test-model-bea800-kserve-5b9646d9bc-gxjkx\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:02.303770 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.303662 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-home\") pod \"sub-test-model-bea800-kserve-5b9646d9bc-gxjkx\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:02.303770 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.303684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13cfc744-f084-44f3-8669-75f34a8eef08-tls-certs\") pod \"sub-test-model-bea800-kserve-5b9646d9bc-gxjkx\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:02.303910 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.303790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj5vx\" (UniqueName: \"kubernetes.io/projected/13cfc744-f084-44f3-8669-75f34a8eef08-kube-api-access-dj5vx\") pod \"sub-test-model-bea800-kserve-5b9646d9bc-gxjkx\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:02.303971 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.303903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-tmp-dir\") pod \"sub-test-model-bea800-kserve-5b9646d9bc-gxjkx\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:02.304122 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.304088 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-home\") pod \"sub-test-model-bea800-kserve-5b9646d9bc-gxjkx\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:02.304278 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.304165 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-model-cache\") pod \"sub-test-model-bea800-kserve-5b9646d9bc-gxjkx\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:02.304278 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.304273 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-tmp-dir\") pod \"sub-test-model-bea800-kserve-5b9646d9bc-gxjkx\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:02.306050 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.306028 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-dshm\") pod \"sub-test-model-bea800-kserve-5b9646d9bc-gxjkx\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:02.306323 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.306307 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13cfc744-f084-44f3-8669-75f34a8eef08-tls-certs\") pod \"sub-test-model-bea800-kserve-5b9646d9bc-gxjkx\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:02.314122 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.314096 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj5vx\" (UniqueName: \"kubernetes.io/projected/13cfc744-f084-44f3-8669-75f34a8eef08-kube-api-access-dj5vx\") pod \"sub-test-model-bea800-kserve-5b9646d9bc-gxjkx\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:02.398163 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.398102 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:02.538330 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:02.538302 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx"] Jul 14 16:05:02.540078 ip-10-0-128-22 kubenswrapper[2579]: W0714 16:05:02.540049 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13cfc744_f084_44f3_8669_75f34a8eef08.slice/crio-15dafbfaf00ff9a2f3bd9ac1066f7bb92c54e25540ea51d6021195e17525e517 WatchSource:0}: Error finding container 15dafbfaf00ff9a2f3bd9ac1066f7bb92c54e25540ea51d6021195e17525e517: Status 404 returned error can't find the container with id 15dafbfaf00ff9a2f3bd9ac1066f7bb92c54e25540ea51d6021195e17525e517 Jul 14 16:05:03.146650 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:03.146607 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" event={"ID":"13cfc744-f084-44f3-8669-75f34a8eef08","Type":"ContainerStarted","Data":"fe1273827d757640d25703191ab3baea34ca3415b5262d707530bbe7676e7007"} Jul 14 16:05:03.147052 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:03.146660 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" event={"ID":"13cfc744-f084-44f3-8669-75f34a8eef08","Type":"ContainerStarted","Data":"15dafbfaf00ff9a2f3bd9ac1066f7bb92c54e25540ea51d6021195e17525e517"} Jul 14 16:05:03.165913 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:03.165863 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" podStartSLOduration=1.165849058 podStartE2EDuration="1.165849058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 16:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 16:05:03.163340891 +0000 UTC m=+2644.015724055" watchObservedRunningTime="2026-07-14 16:05:03.165849058 +0000 UTC m=+2644.018232221" Jul 14 16:05:06.089575 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.089529 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" podUID="8a9faae4-ac8b-4840-bfc7-3729ef9cc56d" containerName="main" containerID="cri-o://c2b9ec5e269426fe0a894f1367b1326d51f3e129ed3337144f018b4225bea701" gracePeriod=45 Jul 14 16:05:06.343137 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.343081 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:05:06.438230 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.438193 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-tmp-dir\") pod \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " Jul 14 16:05:06.438469 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.438260 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cqzt\" (UniqueName: \"kubernetes.io/projected/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-kube-api-access-2cqzt\") pod \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " Jul 14 16:05:06.438469 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.438277 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-model-cache\") pod \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " Jul 14 16:05:06.438469 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.438303 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-tls-certs\") pod \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " Jul 14 16:05:06.438469 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.438330 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-dshm\") pod \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " Jul 14 16:05:06.438469 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.438357 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-home\") pod \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\" (UID: \"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d\") " Jul 14 16:05:06.438469 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.438436 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a9faae4-ac8b-4840-bfc7-3729ef9cc56d" (UID: "8a9faae4-ac8b-4840-bfc7-3729ef9cc56d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:05:06.438764 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.438620 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a9faae4-ac8b-4840-bfc7-3729ef9cc56d" (UID: "8a9faae4-ac8b-4840-bfc7-3729ef9cc56d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:05:06.438764 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.438628 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-tmp-dir\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:05:06.438764 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.438670 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-home" (OuterVolumeSpecName: "home") pod "8a9faae4-ac8b-4840-bfc7-3729ef9cc56d" (UID: "8a9faae4-ac8b-4840-bfc7-3729ef9cc56d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:05:06.440678 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.440649 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-dshm" (OuterVolumeSpecName: "dshm") pod "8a9faae4-ac8b-4840-bfc7-3729ef9cc56d" (UID: "8a9faae4-ac8b-4840-bfc7-3729ef9cc56d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:05:06.440782 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.440686 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a9faae4-ac8b-4840-bfc7-3729ef9cc56d" (UID: "8a9faae4-ac8b-4840-bfc7-3729ef9cc56d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 16:05:06.440782 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.440710 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-kube-api-access-2cqzt" (OuterVolumeSpecName: "kube-api-access-2cqzt") pod "8a9faae4-ac8b-4840-bfc7-3729ef9cc56d" (UID: "8a9faae4-ac8b-4840-bfc7-3729ef9cc56d"). InnerVolumeSpecName "kube-api-access-2cqzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 16:05:06.539857 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.539826 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cqzt\" (UniqueName: \"kubernetes.io/projected/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-kube-api-access-2cqzt\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:05:06.539975 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.539861 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-model-cache\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:05:06.539975 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.539871 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-tls-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:05:06.539975 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.539879 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-dshm\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:05:06.539975 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.539886 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d-home\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:05:06.893883 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:06.893814 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" podUID="cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb" containerName="main" containerID="cri-o://173f06ca7ed6cb51f59e17dd0d65ae4edd018b93b15d2067e59676c3b25d325d" gracePeriod=45 Jul 14 16:05:07.132562 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.132540 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:05:07.168385 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.168309 2579 generic.go:356] "Generic (PLEG): container finished" podID="cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb" containerID="173f06ca7ed6cb51f59e17dd0d65ae4edd018b93b15d2067e59676c3b25d325d" exitCode=0 Jul 14 16:05:07.168518 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.168397 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" Jul 14 16:05:07.168518 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.168438 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" event={"ID":"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb","Type":"ContainerDied","Data":"173f06ca7ed6cb51f59e17dd0d65ae4edd018b93b15d2067e59676c3b25d325d"} Jul 14 16:05:07.168518 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.168484 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c" event={"ID":"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb","Type":"ContainerDied","Data":"da5ef16068775dae4e8c3f99110f40e623d698046103dfacc21b9cb536db6a7b"} Jul 14 16:05:07.168518 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.168509 2579 scope.go:117] "RemoveContainer" containerID="173f06ca7ed6cb51f59e17dd0d65ae4edd018b93b15d2067e59676c3b25d325d" Jul 14 16:05:07.169740 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.169721 2579 generic.go:356] "Generic (PLEG): container finished" podID="8a9faae4-ac8b-4840-bfc7-3729ef9cc56d" containerID="c2b9ec5e269426fe0a894f1367b1326d51f3e129ed3337144f018b4225bea701" exitCode=0 Jul 14 16:05:07.169838 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.169750 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" event={"ID":"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d","Type":"ContainerDied","Data":"c2b9ec5e269426fe0a894f1367b1326d51f3e129ed3337144f018b4225bea701"} Jul 14 16:05:07.169838 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.169770 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" Jul 14 16:05:07.169838 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.169787 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz" event={"ID":"8a9faae4-ac8b-4840-bfc7-3729ef9cc56d","Type":"ContainerDied","Data":"e28a3d30275c222150d1fe89d8b1cd49214a27cca138b57f8cdbfda8642168a9"} Jul 14 16:05:07.178393 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.178325 2579 scope.go:117] "RemoveContainer" containerID="173f06ca7ed6cb51f59e17dd0d65ae4edd018b93b15d2067e59676c3b25d325d" Jul 14 16:05:07.178689 ip-10-0-128-22 kubenswrapper[2579]: E0714 16:05:07.178662 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"173f06ca7ed6cb51f59e17dd0d65ae4edd018b93b15d2067e59676c3b25d325d\": container with ID starting with 173f06ca7ed6cb51f59e17dd0d65ae4edd018b93b15d2067e59676c3b25d325d not found: ID does not exist" containerID="173f06ca7ed6cb51f59e17dd0d65ae4edd018b93b15d2067e59676c3b25d325d" Jul 14 16:05:07.178794 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.178689 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"173f06ca7ed6cb51f59e17dd0d65ae4edd018b93b15d2067e59676c3b25d325d"} err="failed to get container status \"173f06ca7ed6cb51f59e17dd0d65ae4edd018b93b15d2067e59676c3b25d325d\": rpc error: code = NotFound desc = could not find container \"173f06ca7ed6cb51f59e17dd0d65ae4edd018b93b15d2067e59676c3b25d325d\": container with ID starting with 173f06ca7ed6cb51f59e17dd0d65ae4edd018b93b15d2067e59676c3b25d325d not found: ID does not exist" Jul 14 16:05:07.178794 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.178706 2579 scope.go:117] "RemoveContainer" containerID="c2b9ec5e269426fe0a894f1367b1326d51f3e129ed3337144f018b4225bea701" Jul 14 16:05:07.186652 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.186637 2579 scope.go:117] "RemoveContainer" containerID="c2b9ec5e269426fe0a894f1367b1326d51f3e129ed3337144f018b4225bea701" Jul 14 16:05:07.186892 ip-10-0-128-22 kubenswrapper[2579]: E0714 16:05:07.186873 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b9ec5e269426fe0a894f1367b1326d51f3e129ed3337144f018b4225bea701\": container with ID starting with c2b9ec5e269426fe0a894f1367b1326d51f3e129ed3337144f018b4225bea701 not found: ID does not exist" containerID="c2b9ec5e269426fe0a894f1367b1326d51f3e129ed3337144f018b4225bea701" Jul 14 16:05:07.186955 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.186896 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b9ec5e269426fe0a894f1367b1326d51f3e129ed3337144f018b4225bea701"} err="failed to get container status \"c2b9ec5e269426fe0a894f1367b1326d51f3e129ed3337144f018b4225bea701\": rpc error: code = NotFound desc = could not find container \"c2b9ec5e269426fe0a894f1367b1326d51f3e129ed3337144f018b4225bea701\": container with ID starting with c2b9ec5e269426fe0a894f1367b1326d51f3e129ed3337144f018b4225bea701 not found: ID does not exist" Jul 14 16:05:07.193907 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.193886 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz"] Jul 14 16:05:07.196115 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.196092 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bea800/auth-test-model-bea800-kserve-6ddb7566c9-kv7qz"] Jul 14 16:05:07.245819 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.245796 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-dshm\") pod \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " Jul 14 16:05:07.245898 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.245838 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-tmp-dir\") pod \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " Jul 14 16:05:07.245898 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.245884 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-model-cache\") pod \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " Jul 14 16:05:07.245967 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.245914 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjmtz\" (UniqueName: \"kubernetes.io/projected/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-kube-api-access-sjmtz\") pod \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " Jul 14 16:05:07.245967 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.245929 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-home\") pod \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " Jul 14 16:05:07.246063 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.245965 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-tls-certs\") pod \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\" (UID: \"cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb\") " Jul 14 16:05:07.246228 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.246202 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb" (UID: "cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:05:07.246308 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.246229 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb" (UID: "cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:05:07.246308 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.246238 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-home" (OuterVolumeSpecName: "home") pod "cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb" (UID: "cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:05:07.248062 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.248042 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-dshm" (OuterVolumeSpecName: "dshm") pod "cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb" (UID: "cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:05:07.248210 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.248189 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb" (UID: "cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 16:05:07.248278 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.248262 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-kube-api-access-sjmtz" (OuterVolumeSpecName: "kube-api-access-sjmtz") pod "cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb" (UID: "cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb"). InnerVolumeSpecName "kube-api-access-sjmtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 16:05:07.347465 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.347438 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjmtz\" (UniqueName: \"kubernetes.io/projected/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-kube-api-access-sjmtz\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:05:07.347465 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.347460 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-home\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:05:07.347597 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.347469 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-tls-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:05:07.347597 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.347478 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-dshm\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:05:07.347597 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.347485 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-tmp-dir\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:05:07.347597 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.347492 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb-model-cache\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:05:07.492597 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.492572 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c"] Jul 14 16:05:07.494723 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.494707 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-519447/auth-test-model-519447-kserve-5974c9979-chx7c"] Jul 14 16:05:07.696489 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.696460 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a9faae4-ac8b-4840-bfc7-3729ef9cc56d" path="/var/lib/kubelet/pods/8a9faae4-ac8b-4840-bfc7-3729ef9cc56d/volumes" Jul 14 16:05:07.696882 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:07.696868 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb" path="/var/lib/kubelet/pods/cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb/volumes" Jul 14 16:05:12.398618 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:12.398568 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:12.399061 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:12.398654 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:12.410902 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:12.410874 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:13.217085 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:13.217055 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:05:16.292641 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:16.292604 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:05:23.979148 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:23.979107 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc"] Jul 14 16:05:23.979538 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:23.979476 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9faae4-ac8b-4840-bfc7-3729ef9cc56d" containerName="main" Jul 14 16:05:23.979538 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:23.979488 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9faae4-ac8b-4840-bfc7-3729ef9cc56d" containerName="main" Jul 14 16:05:23.979538 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:23.979507 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb" containerName="main" Jul 14 16:05:23.979538 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:23.979513 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb" containerName="main" Jul 14 16:05:23.979665 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:23.979586 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc60343b-bc6b-4a2a-8ea1-1be93bc44bfb" containerName="main" Jul 14 16:05:23.979665 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:23.979598 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a9faae4-ac8b-4840-bfc7-3729ef9cc56d" containerName="main" Jul 14 16:05:23.983767 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:23.983748 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:23.988440 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:23.988415 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-519447\"/\"sub-test-model-519447-kserve-self-signed-certs\"" Jul 14 16:05:23.988674 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:23.988497 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-519447\"/\"default-dockercfg-r57n8\"" Jul 14 16:05:23.988771 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:23.988516 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-519447\"/\"kube-root-ca.crt\"" Jul 14 16:05:23.988771 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:23.988656 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-519447\"/\"openshift-service-ca.crt\"" Jul 14 16:05:23.992416 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:23.992391 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc"] Jul 14 16:05:24.010355 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.010313 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:05:24.072630 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.072601 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-dshm\") pod \"sub-test-model-519447-kserve-6fc68c8cf-gsbqc\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:24.072732 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.072645 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2abae131-0461-4ac4-a301-3e551833105e-tls-certs\") pod \"sub-test-model-519447-kserve-6fc68c8cf-gsbqc\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:24.072732 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.072678 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-tmp-dir\") pod \"sub-test-model-519447-kserve-6fc68c8cf-gsbqc\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:24.072848 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.072748 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-model-cache\") pod \"sub-test-model-519447-kserve-6fc68c8cf-gsbqc\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:24.072848 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.072774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwmjk\" (UniqueName: \"kubernetes.io/projected/2abae131-0461-4ac4-a301-3e551833105e-kube-api-access-bwmjk\") pod \"sub-test-model-519447-kserve-6fc68c8cf-gsbqc\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:24.072848 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.072816 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-home\") pod \"sub-test-model-519447-kserve-6fc68c8cf-gsbqc\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:24.173176 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.173152 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2abae131-0461-4ac4-a301-3e551833105e-tls-certs\") pod \"sub-test-model-519447-kserve-6fc68c8cf-gsbqc\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:24.173270 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.173206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-tmp-dir\") pod \"sub-test-model-519447-kserve-6fc68c8cf-gsbqc\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:24.173270 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.173264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-model-cache\") pod \"sub-test-model-519447-kserve-6fc68c8cf-gsbqc\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:24.173369 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.173288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwmjk\" (UniqueName: \"kubernetes.io/projected/2abae131-0461-4ac4-a301-3e551833105e-kube-api-access-bwmjk\") pod \"sub-test-model-519447-kserve-6fc68c8cf-gsbqc\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:24.173369 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.173325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-home\") pod \"sub-test-model-519447-kserve-6fc68c8cf-gsbqc\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:24.173465 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.173378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-dshm\") pod \"sub-test-model-519447-kserve-6fc68c8cf-gsbqc\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:24.173751 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.173729 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-tmp-dir\") pod \"sub-test-model-519447-kserve-6fc68c8cf-gsbqc\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:24.173841 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.173780 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-model-cache\") pod \"sub-test-model-519447-kserve-6fc68c8cf-gsbqc\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:24.173841 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.173794 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-home\") pod \"sub-test-model-519447-kserve-6fc68c8cf-gsbqc\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:24.175514 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.175493 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-dshm\") pod \"sub-test-model-519447-kserve-6fc68c8cf-gsbqc\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:24.175751 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.175736 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2abae131-0461-4ac4-a301-3e551833105e-tls-certs\") pod \"sub-test-model-519447-kserve-6fc68c8cf-gsbqc\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:24.183371 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.183353 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwmjk\" (UniqueName: \"kubernetes.io/projected/2abae131-0461-4ac4-a301-3e551833105e-kube-api-access-bwmjk\") pod \"sub-test-model-519447-kserve-6fc68c8cf-gsbqc\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:24.294198 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.294117 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:24.428999 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:24.428970 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc"] Jul 14 16:05:24.430160 ip-10-0-128-22 kubenswrapper[2579]: W0714 16:05:24.430132 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2abae131_0461_4ac4_a301_3e551833105e.slice/crio-a4008a0bd427e5f2b06f263853f14d19afae43147d9c1e13bded66b39b7d2658 WatchSource:0}: Error finding container a4008a0bd427e5f2b06f263853f14d19afae43147d9c1e13bded66b39b7d2658: Status 404 returned error can't find the container with id a4008a0bd427e5f2b06f263853f14d19afae43147d9c1e13bded66b39b7d2658 Jul 14 16:05:25.266669 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:25.266628 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" event={"ID":"2abae131-0461-4ac4-a301-3e551833105e","Type":"ContainerStarted","Data":"ff8088565a7890998db907256f6a9221616df8fbe1340e52734aca7d5e2f47e1"} Jul 14 16:05:25.266669 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:25.266681 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" event={"ID":"2abae131-0461-4ac4-a301-3e551833105e","Type":"ContainerStarted","Data":"a4008a0bd427e5f2b06f263853f14d19afae43147d9c1e13bded66b39b7d2658"} Jul 14 16:05:25.284767 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:25.284715 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" podStartSLOduration=2.284701767 podStartE2EDuration="2.284701767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 16:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 16:05:25.28300133 +0000 UTC m=+2666.135384496" watchObservedRunningTime="2026-07-14 16:05:25.284701767 +0000 UTC m=+2666.137084954" Jul 14 16:05:27.406716 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:27.406683 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:05:34.295033 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:34.294993 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:34.295033 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:34.295049 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:34.307363 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:34.307341 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:34.338210 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:34.338188 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:05:52.702233 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:52.702144 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:05:57.920994 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:57.920956 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx"] Jul 14 16:05:57.944428 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:57.944395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:05:58.870589 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:58.870557 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc"] Jul 14 16:05:58.890611 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:58.890564 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:05:59.862242 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:59.862215 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 16:05:59.870610 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:05:59.870587 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 16:06:10.602513 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:10.602475 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:06:12.941664 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:12.941615 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" podUID="13cfc744-f084-44f3-8669-75f34a8eef08" containerName="main" containerID="cri-o://fe1273827d757640d25703191ab3baea34ca3415b5262d707530bbe7676e7007" gracePeriod=45 Jul 14 16:06:13.194774 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.194710 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:06:13.272303 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.272268 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-dshm\") pod \"13cfc744-f084-44f3-8669-75f34a8eef08\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " Jul 14 16:06:13.272464 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.272309 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-model-cache\") pod \"13cfc744-f084-44f3-8669-75f34a8eef08\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " Jul 14 16:06:13.272464 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.272368 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dj5vx\" (UniqueName: \"kubernetes.io/projected/13cfc744-f084-44f3-8669-75f34a8eef08-kube-api-access-dj5vx\") pod \"13cfc744-f084-44f3-8669-75f34a8eef08\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " Jul 14 16:06:13.272464 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.272399 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-tmp-dir\") pod \"13cfc744-f084-44f3-8669-75f34a8eef08\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " Jul 14 16:06:13.272464 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.272423 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-home\") pod \"13cfc744-f084-44f3-8669-75f34a8eef08\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " Jul 14 16:06:13.272668 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.272499 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13cfc744-f084-44f3-8669-75f34a8eef08-tls-certs\") pod \"13cfc744-f084-44f3-8669-75f34a8eef08\" (UID: \"13cfc744-f084-44f3-8669-75f34a8eef08\") " Jul 14 16:06:13.272668 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.272597 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-model-cache" (OuterVolumeSpecName: "model-cache") pod "13cfc744-f084-44f3-8669-75f34a8eef08" (UID: "13cfc744-f084-44f3-8669-75f34a8eef08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:06:13.272754 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.272650 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13cfc744-f084-44f3-8669-75f34a8eef08" (UID: "13cfc744-f084-44f3-8669-75f34a8eef08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:06:13.272824 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.272732 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-home" (OuterVolumeSpecName: "home") pod "13cfc744-f084-44f3-8669-75f34a8eef08" (UID: "13cfc744-f084-44f3-8669-75f34a8eef08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:06:13.272882 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.272863 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-model-cache\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:06:13.272882 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.272877 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-tmp-dir\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:06:13.272982 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.272884 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-home\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:06:13.274657 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.274620 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13cfc744-f084-44f3-8669-75f34a8eef08-kube-api-access-dj5vx" (OuterVolumeSpecName: "kube-api-access-dj5vx") pod "13cfc744-f084-44f3-8669-75f34a8eef08" (UID: "13cfc744-f084-44f3-8669-75f34a8eef08"). InnerVolumeSpecName "kube-api-access-dj5vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 16:06:13.274770 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.274691 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-dshm" (OuterVolumeSpecName: "dshm") pod "13cfc744-f084-44f3-8669-75f34a8eef08" (UID: "13cfc744-f084-44f3-8669-75f34a8eef08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:06:13.274770 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.274693 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13cfc744-f084-44f3-8669-75f34a8eef08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13cfc744-f084-44f3-8669-75f34a8eef08" (UID: "13cfc744-f084-44f3-8669-75f34a8eef08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 16:06:13.373720 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.373692 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13cfc744-f084-44f3-8669-75f34a8eef08-dshm\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:06:13.373720 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.373719 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj5vx\" (UniqueName: \"kubernetes.io/projected/13cfc744-f084-44f3-8669-75f34a8eef08-kube-api-access-dj5vx\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:06:13.373904 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.373730 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13cfc744-f084-44f3-8669-75f34a8eef08-tls-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:06:13.540564 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.540476 2579 generic.go:356] "Generic (PLEG): container finished" podID="13cfc744-f084-44f3-8669-75f34a8eef08" containerID="fe1273827d757640d25703191ab3baea34ca3415b5262d707530bbe7676e7007" exitCode=0 Jul 14 16:06:13.540564 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.540527 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" Jul 14 16:06:13.540564 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.540526 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" event={"ID":"13cfc744-f084-44f3-8669-75f34a8eef08","Type":"ContainerDied","Data":"fe1273827d757640d25703191ab3baea34ca3415b5262d707530bbe7676e7007"} Jul 14 16:06:13.540836 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.540578 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx" event={"ID":"13cfc744-f084-44f3-8669-75f34a8eef08","Type":"ContainerDied","Data":"15dafbfaf00ff9a2f3bd9ac1066f7bb92c54e25540ea51d6021195e17525e517"} Jul 14 16:06:13.540836 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.540602 2579 scope.go:117] "RemoveContainer" containerID="fe1273827d757640d25703191ab3baea34ca3415b5262d707530bbe7676e7007" Jul 14 16:06:13.550374 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.550359 2579 scope.go:117] "RemoveContainer" containerID="fe1273827d757640d25703191ab3baea34ca3415b5262d707530bbe7676e7007" Jul 14 16:06:13.550594 ip-10-0-128-22 kubenswrapper[2579]: E0714 16:06:13.550575 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe1273827d757640d25703191ab3baea34ca3415b5262d707530bbe7676e7007\": container with ID starting with fe1273827d757640d25703191ab3baea34ca3415b5262d707530bbe7676e7007 not found: ID does not exist" containerID="fe1273827d757640d25703191ab3baea34ca3415b5262d707530bbe7676e7007" Jul 14 16:06:13.550667 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.550606 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe1273827d757640d25703191ab3baea34ca3415b5262d707530bbe7676e7007"} err="failed to get container status \"fe1273827d757640d25703191ab3baea34ca3415b5262d707530bbe7676e7007\": rpc error: code = NotFound desc = could not find container \"fe1273827d757640d25703191ab3baea34ca3415b5262d707530bbe7676e7007\": container with ID starting with fe1273827d757640d25703191ab3baea34ca3415b5262d707530bbe7676e7007 not found: ID does not exist" Jul 14 16:06:13.563920 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.563898 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx"] Jul 14 16:06:13.568424 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.568402 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bea800/sub-test-model-bea800-kserve-5b9646d9bc-gxjkx"] Jul 14 16:06:13.697706 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.697664 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13cfc744-f084-44f3-8669-75f34a8eef08" path="/var/lib/kubelet/pods/13cfc744-f084-44f3-8669-75f34a8eef08/volumes" Jul 14 16:06:13.891624 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:13.891585 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" podUID="2abae131-0461-4ac4-a301-3e551833105e" containerName="main" containerID="cri-o://ff8088565a7890998db907256f6a9221616df8fbe1340e52734aca7d5e2f47e1" gracePeriod=45 Jul 14 16:06:14.139059 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.139032 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:06:14.181735 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.181673 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-tmp-dir\") pod \"2abae131-0461-4ac4-a301-3e551833105e\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " Jul 14 16:06:14.181735 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.181713 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2abae131-0461-4ac4-a301-3e551833105e-tls-certs\") pod \"2abae131-0461-4ac4-a301-3e551833105e\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " Jul 14 16:06:14.181897 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.181750 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-model-cache\") pod \"2abae131-0461-4ac4-a301-3e551833105e\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " Jul 14 16:06:14.181897 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.181769 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwmjk\" (UniqueName: \"kubernetes.io/projected/2abae131-0461-4ac4-a301-3e551833105e-kube-api-access-bwmjk\") pod \"2abae131-0461-4ac4-a301-3e551833105e\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " Jul 14 16:06:14.181897 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.181803 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-home\") pod \"2abae131-0461-4ac4-a301-3e551833105e\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " Jul 14 16:06:14.181897 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.181882 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-dshm\") pod \"2abae131-0461-4ac4-a301-3e551833105e\" (UID: \"2abae131-0461-4ac4-a301-3e551833105e\") " Jul 14 16:06:14.182077 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.181966 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2abae131-0461-4ac4-a301-3e551833105e" (UID: "2abae131-0461-4ac4-a301-3e551833105e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:06:14.182077 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.182022 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-model-cache" (OuterVolumeSpecName: "model-cache") pod "2abae131-0461-4ac4-a301-3e551833105e" (UID: "2abae131-0461-4ac4-a301-3e551833105e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:06:14.182221 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.182122 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-home" (OuterVolumeSpecName: "home") pod "2abae131-0461-4ac4-a301-3e551833105e" (UID: "2abae131-0461-4ac4-a301-3e551833105e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:06:14.182285 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.182258 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-tmp-dir\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:06:14.182285 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.182272 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-model-cache\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:06:14.182285 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.182282 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-home\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:06:14.184008 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.183983 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-dshm" (OuterVolumeSpecName: "dshm") pod "2abae131-0461-4ac4-a301-3e551833105e" (UID: "2abae131-0461-4ac4-a301-3e551833105e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:06:14.184008 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.183992 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abae131-0461-4ac4-a301-3e551833105e-kube-api-access-bwmjk" (OuterVolumeSpecName: "kube-api-access-bwmjk") pod "2abae131-0461-4ac4-a301-3e551833105e" (UID: "2abae131-0461-4ac4-a301-3e551833105e"). InnerVolumeSpecName "kube-api-access-bwmjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 16:06:14.184147 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.184022 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2abae131-0461-4ac4-a301-3e551833105e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2abae131-0461-4ac4-a301-3e551833105e" (UID: "2abae131-0461-4ac4-a301-3e551833105e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 16:06:14.282948 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.282925 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2abae131-0461-4ac4-a301-3e551833105e-tls-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:06:14.282948 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.282949 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwmjk\" (UniqueName: \"kubernetes.io/projected/2abae131-0461-4ac4-a301-3e551833105e-kube-api-access-bwmjk\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:06:14.283102 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.282958 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2abae131-0461-4ac4-a301-3e551833105e-dshm\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:06:14.548815 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.548727 2579 generic.go:356] "Generic (PLEG): container finished" podID="2abae131-0461-4ac4-a301-3e551833105e" containerID="ff8088565a7890998db907256f6a9221616df8fbe1340e52734aca7d5e2f47e1" exitCode=0 Jul 14 16:06:14.548815 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.548777 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" Jul 14 16:06:14.549006 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.548775 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" event={"ID":"2abae131-0461-4ac4-a301-3e551833105e","Type":"ContainerDied","Data":"ff8088565a7890998db907256f6a9221616df8fbe1340e52734aca7d5e2f47e1"} Jul 14 16:06:14.549006 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.548880 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc" event={"ID":"2abae131-0461-4ac4-a301-3e551833105e","Type":"ContainerDied","Data":"a4008a0bd427e5f2b06f263853f14d19afae43147d9c1e13bded66b39b7d2658"} Jul 14 16:06:14.549006 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.548904 2579 scope.go:117] "RemoveContainer" containerID="ff8088565a7890998db907256f6a9221616df8fbe1340e52734aca7d5e2f47e1" Jul 14 16:06:14.558463 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.558446 2579 scope.go:117] "RemoveContainer" containerID="ff8088565a7890998db907256f6a9221616df8fbe1340e52734aca7d5e2f47e1" Jul 14 16:06:14.558718 ip-10-0-128-22 kubenswrapper[2579]: E0714 16:06:14.558698 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff8088565a7890998db907256f6a9221616df8fbe1340e52734aca7d5e2f47e1\": container with ID starting with ff8088565a7890998db907256f6a9221616df8fbe1340e52734aca7d5e2f47e1 not found: ID does not exist" containerID="ff8088565a7890998db907256f6a9221616df8fbe1340e52734aca7d5e2f47e1" Jul 14 16:06:14.558783 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.558730 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff8088565a7890998db907256f6a9221616df8fbe1340e52734aca7d5e2f47e1"} err="failed to get container status \"ff8088565a7890998db907256f6a9221616df8fbe1340e52734aca7d5e2f47e1\": rpc error: code = NotFound desc = could not find container \"ff8088565a7890998db907256f6a9221616df8fbe1340e52734aca7d5e2f47e1\": container with ID starting with ff8088565a7890998db907256f6a9221616df8fbe1340e52734aca7d5e2f47e1 not found: ID does not exist" Jul 14 16:06:14.571614 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.571592 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc"] Jul 14 16:06:14.575268 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:14.575246 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-519447/sub-test-model-519447-kserve-6fc68c8cf-gsbqc"] Jul 14 16:06:15.696430 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:15.696397 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2abae131-0461-4ac4-a301-3e551833105e" path="/var/lib/kubelet/pods/2abae131-0461-4ac4-a301-3e551833105e/volumes" Jul 14 16:06:19.709582 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.709542 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg"] Jul 14 16:06:19.710039 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.709991 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13cfc744-f084-44f3-8669-75f34a8eef08" containerName="main" Jul 14 16:06:19.710039 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.710010 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="13cfc744-f084-44f3-8669-75f34a8eef08" containerName="main" Jul 14 16:06:19.710039 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.710029 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2abae131-0461-4ac4-a301-3e551833105e" containerName="main" Jul 14 16:06:19.710039 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.710038 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2abae131-0461-4ac4-a301-3e551833105e" containerName="main" Jul 14 16:06:19.710300 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.710146 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2abae131-0461-4ac4-a301-3e551833105e" containerName="main" Jul 14 16:06:19.710300 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.710159 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="13cfc744-f084-44f3-8669-75f34a8eef08" containerName="main" Jul 14 16:06:19.714200 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.714163 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:19.718534 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.718512 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bea800\"/\"openshift-service-ca.crt\"" Jul 14 16:06:19.719431 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.719414 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bea800\"/\"kube-root-ca.crt\"" Jul 14 16:06:19.719528 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.719429 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bea800\"/\"default-dockercfg-c7tgc\"" Jul 14 16:06:19.720204 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.720187 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bea800\"/\"rate-test-model-bea800-kserve-self-signed-certs\"" Jul 14 16:06:19.729492 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.729457 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg"] Jul 14 16:06:19.817865 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.817831 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:06:19.825823 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.825790 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4kd5\" (UniqueName: \"kubernetes.io/projected/f04682d2-df47-4e81-a179-a9c359815e1e-kube-api-access-g4kd5\") pod \"rate-test-model-bea800-kserve-66df66fcd7-vrntg\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:19.825912 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.825845 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-dshm\") pod \"rate-test-model-bea800-kserve-66df66fcd7-vrntg\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:19.825912 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.825870 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-home\") pod \"rate-test-model-bea800-kserve-66df66fcd7-vrntg\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:19.825912 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.825905 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-model-cache\") pod \"rate-test-model-bea800-kserve-66df66fcd7-vrntg\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:19.826033 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.825920 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-tmp-dir\") pod \"rate-test-model-bea800-kserve-66df66fcd7-vrntg\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:19.826033 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.825983 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f04682d2-df47-4e81-a179-a9c359815e1e-tls-certs\") pod \"rate-test-model-bea800-kserve-66df66fcd7-vrntg\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:19.926727 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.926694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-dshm\") pod \"rate-test-model-bea800-kserve-66df66fcd7-vrntg\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:19.926727 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.926731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-home\") pod \"rate-test-model-bea800-kserve-66df66fcd7-vrntg\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:19.926932 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.926769 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-model-cache\") pod \"rate-test-model-bea800-kserve-66df66fcd7-vrntg\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:19.926932 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.926785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-tmp-dir\") pod \"rate-test-model-bea800-kserve-66df66fcd7-vrntg\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:19.926932 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.926810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f04682d2-df47-4e81-a179-a9c359815e1e-tls-certs\") pod \"rate-test-model-bea800-kserve-66df66fcd7-vrntg\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:19.926932 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.926841 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4kd5\" (UniqueName: \"kubernetes.io/projected/f04682d2-df47-4e81-a179-a9c359815e1e-kube-api-access-g4kd5\") pod \"rate-test-model-bea800-kserve-66df66fcd7-vrntg\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:19.927240 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.927210 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-model-cache\") pod \"rate-test-model-bea800-kserve-66df66fcd7-vrntg\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:19.927240 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.927235 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-home\") pod \"rate-test-model-bea800-kserve-66df66fcd7-vrntg\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:19.927382 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.927292 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-tmp-dir\") pod \"rate-test-model-bea800-kserve-66df66fcd7-vrntg\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:19.928932 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.928911 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-dshm\") pod \"rate-test-model-bea800-kserve-66df66fcd7-vrntg\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:19.929337 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.929317 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f04682d2-df47-4e81-a179-a9c359815e1e-tls-certs\") pod \"rate-test-model-bea800-kserve-66df66fcd7-vrntg\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:19.937896 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:19.937874 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4kd5\" (UniqueName: \"kubernetes.io/projected/f04682d2-df47-4e81-a179-a9c359815e1e-kube-api-access-g4kd5\") pod \"rate-test-model-bea800-kserve-66df66fcd7-vrntg\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:20.023680 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:20.023624 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:20.156368 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:20.156340 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg"] Jul 14 16:06:20.157775 ip-10-0-128-22 kubenswrapper[2579]: W0714 16:06:20.157739 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf04682d2_df47_4e81_a179_a9c359815e1e.slice/crio-16ca995876da8e3aea59a77450e43ae8e128db8d1426f2e5eaeb692f3eb9b1d2 WatchSource:0}: Error finding container 16ca995876da8e3aea59a77450e43ae8e128db8d1426f2e5eaeb692f3eb9b1d2: Status 404 returned error can't find the container with id 16ca995876da8e3aea59a77450e43ae8e128db8d1426f2e5eaeb692f3eb9b1d2 Jul 14 16:06:20.580581 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:20.580537 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" event={"ID":"f04682d2-df47-4e81-a179-a9c359815e1e","Type":"ContainerStarted","Data":"16e30b8ba7cdc40c89777f7607e246e1239a47781dd329b39d9dbb1728925cae"} Jul 14 16:06:20.580749 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:20.580592 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" event={"ID":"f04682d2-df47-4e81-a179-a9c359815e1e","Type":"ContainerStarted","Data":"16ca995876da8e3aea59a77450e43ae8e128db8d1426f2e5eaeb692f3eb9b1d2"} Jul 14 16:06:20.601836 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:20.601788 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" podStartSLOduration=1.601774266 podStartE2EDuration="1.601774266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 16:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 16:06:20.600467811 +0000 UTC m=+2721.452850976" watchObservedRunningTime="2026-07-14 16:06:20.601774266 +0000 UTC m=+2721.454157430" Jul 14 16:06:25.402817 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:25.402781 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:06:30.024762 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:30.024725 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:30.025209 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:30.024812 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:30.037512 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:30.037489 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:30.649966 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:30.649935 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:06:43.618864 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:43.618802 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:06:44.946068 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:44.946016 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:06:46.281647 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.281609 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2"] Jul 14 16:06:46.285058 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.285036 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:46.288117 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.288089 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-519447\"/\"rate-test-model-519447-kserve-self-signed-certs\"" Jul 14 16:06:46.289493 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.289473 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-519447\"/\"kube-root-ca.crt\"" Jul 14 16:06:46.289763 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.289727 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-519447\"/\"default-dockercfg-r57n8\"" Jul 14 16:06:46.289851 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.289794 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-519447\"/\"openshift-service-ca.crt\"" Jul 14 16:06:46.295757 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.295725 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2"] Jul 14 16:06:46.311373 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.311334 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:06:46.459152 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.459102 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckrw4\" (UniqueName: \"kubernetes.io/projected/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-kube-api-access-ckrw4\") pod \"rate-test-model-519447-kserve-5c9fd96c9f-cllv2\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:46.459152 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.459185 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-dshm\") pod \"rate-test-model-519447-kserve-5c9fd96c9f-cllv2\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:46.459407 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.459243 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-home\") pod \"rate-test-model-519447-kserve-5c9fd96c9f-cllv2\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:46.459407 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.459288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-tls-certs\") pod \"rate-test-model-519447-kserve-5c9fd96c9f-cllv2\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:46.459407 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.459310 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-model-cache\") pod \"rate-test-model-519447-kserve-5c9fd96c9f-cllv2\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:46.459407 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.459335 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-tmp-dir\") pod \"rate-test-model-519447-kserve-5c9fd96c9f-cllv2\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:46.560040 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.559934 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-dshm\") pod \"rate-test-model-519447-kserve-5c9fd96c9f-cllv2\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:46.560040 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.559990 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-home\") pod \"rate-test-model-519447-kserve-5c9fd96c9f-cllv2\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:46.560040 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.560018 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-tls-certs\") pod \"rate-test-model-519447-kserve-5c9fd96c9f-cllv2\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:46.560040 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.560038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-model-cache\") pod \"rate-test-model-519447-kserve-5c9fd96c9f-cllv2\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:46.560402 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.560061 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-tmp-dir\") pod \"rate-test-model-519447-kserve-5c9fd96c9f-cllv2\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:46.560402 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.560166 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckrw4\" (UniqueName: \"kubernetes.io/projected/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-kube-api-access-ckrw4\") pod \"rate-test-model-519447-kserve-5c9fd96c9f-cllv2\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:46.560536 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.560505 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-home\") pod \"rate-test-model-519447-kserve-5c9fd96c9f-cllv2\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:46.560645 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.560545 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-model-cache\") pod \"rate-test-model-519447-kserve-5c9fd96c9f-cllv2\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:46.560645 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.560583 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-tmp-dir\") pod \"rate-test-model-519447-kserve-5c9fd96c9f-cllv2\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:46.562683 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.562660 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-dshm\") pod \"rate-test-model-519447-kserve-5c9fd96c9f-cllv2\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:46.562916 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.562893 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-tls-certs\") pod \"rate-test-model-519447-kserve-5c9fd96c9f-cllv2\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:46.569786 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.569753 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckrw4\" (UniqueName: \"kubernetes.io/projected/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-kube-api-access-ckrw4\") pod \"rate-test-model-519447-kserve-5c9fd96c9f-cllv2\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:46.596594 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.596545 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:46.743630 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:46.743599 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2"] Jul 14 16:06:46.745042 ip-10-0-128-22 kubenswrapper[2579]: W0714 16:06:46.745006 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25c848bb_84cd_4f57_b2d0_1aaf1c54ecac.slice/crio-540e8a2471ab3ae6a2da1dcc3ff967b0c8609d922f72dcdc3e3ca78699797bf0 WatchSource:0}: Error finding container 540e8a2471ab3ae6a2da1dcc3ff967b0c8609d922f72dcdc3e3ca78699797bf0: Status 404 returned error can't find the container with id 540e8a2471ab3ae6a2da1dcc3ff967b0c8609d922f72dcdc3e3ca78699797bf0 Jul 14 16:06:47.731632 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:47.731600 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" event={"ID":"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac","Type":"ContainerStarted","Data":"a499f5b08fbbaea5f5d8a3f5d8750593bb99c96ffc79f11c7bc015a961fd543d"} Jul 14 16:06:47.731632 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:47.731639 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" event={"ID":"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac","Type":"ContainerStarted","Data":"540e8a2471ab3ae6a2da1dcc3ff967b0c8609d922f72dcdc3e3ca78699797bf0"} Jul 14 16:06:47.753633 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:47.753583 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" podStartSLOduration=1.753563244 podStartE2EDuration="1.753563244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 16:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 16:06:47.750613915 +0000 UTC m=+2748.602997079" watchObservedRunningTime="2026-07-14 16:06:47.753563244 +0000 UTC m=+2748.605946422" Jul 14 16:06:50.900421 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:50.900380 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:06:51.783800 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:51.783746 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:06:51.886482 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:51.886443 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:06:56.597295 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:56.597258 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:56.597817 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:56.597534 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:56.610195 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:56.610157 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:06:56.790846 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:06:56.790811 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:07:20.396834 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:20.396796 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:07:34.595678 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:34.595640 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:07:41.279942 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:41.279905 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg"] Jul 14 16:07:41.305086 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:41.305054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:07:42.216671 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:42.216636 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2"] Jul 14 16:07:42.230773 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:42.230732 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:07:43.277366 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:43.277324 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7336-140624936488240"] Jul 14 16:07:43.280995 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:43.280976 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936488240" Jul 14 16:07:43.284809 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:43.284774 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-vp495\"" Jul 14 16:07:43.286307 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:43.286283 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7336-140624936488240"] Jul 14 16:07:43.362406 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:43.362367 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg79f\" (UniqueName: \"kubernetes.io/projected/e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a-kube-api-access-dg79f\") pod \"test-curl-7336-140624936488240\" (UID: \"e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a\") " pod="openshift-ingress/test-curl-7336-140624936488240" Jul 14 16:07:43.462958 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:43.462915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg79f\" (UniqueName: \"kubernetes.io/projected/e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a-kube-api-access-dg79f\") pod \"test-curl-7336-140624936488240\" (UID: \"e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a\") " pod="openshift-ingress/test-curl-7336-140624936488240" Jul 14 16:07:43.477745 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:43.477719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg79f\" (UniqueName: \"kubernetes.io/projected/e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a-kube-api-access-dg79f\") pod \"test-curl-7336-140624936488240\" (UID: \"e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a\") " pod="openshift-ingress/test-curl-7336-140624936488240" Jul 14 16:07:43.591607 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:43.591569 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936488240" Jul 14 16:07:43.733698 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:43.733672 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7336-140624936488240"] Jul 14 16:07:43.735634 ip-10-0-128-22 kubenswrapper[2579]: W0714 16:07:43.735604 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9449b85_a3dd_4f12_8d7f_728cbd0a3a3a.slice/crio-fbfb34d59ea54f1dff3d5df0ab40f9b4cb8328406d62b7f5fedc7113b89f4e13 WatchSource:0}: Error finding container fbfb34d59ea54f1dff3d5df0ab40f9b4cb8328406d62b7f5fedc7113b89f4e13: Status 404 returned error can't find the container with id fbfb34d59ea54f1dff3d5df0ab40f9b4cb8328406d62b7f5fedc7113b89f4e13 Jul 14 16:07:43.737302 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:43.737283 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 16:07:44.032105 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:44.031996 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7336-140624936488240" event={"ID":"e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a","Type":"ContainerStarted","Data":"fbfb34d59ea54f1dff3d5df0ab40f9b4cb8328406d62b7f5fedc7113b89f4e13"} Jul 14 16:07:45.039077 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:45.039039 2579 generic.go:356] "Generic (PLEG): container finished" podID="e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a" containerID="0ed2f5e9304d583b2cd55920318e3386b2a5b9b0a5060f18209cda0e6ec53a01" exitCode=0 Jul 14 16:07:45.039487 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:45.039083 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7336-140624936488240" event={"ID":"e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a","Type":"ContainerDied","Data":"0ed2f5e9304d583b2cd55920318e3386b2a5b9b0a5060f18209cda0e6ec53a01"} Jul 14 16:07:46.189526 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:46.189503 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936488240" Jul 14 16:07:46.217139 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:46.217113 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7336-140624936488240_e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a/test-curl-7336-140624936488240/0.log" Jul 14 16:07:46.246542 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:46.246502 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7336-140624936488240"] Jul 14 16:07:46.254224 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:46.254194 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7336-140624936488240"] Jul 14 16:07:46.284865 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:46.284833 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dg79f\" (UniqueName: \"kubernetes.io/projected/e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a-kube-api-access-dg79f\") pod \"e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a\" (UID: \"e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a\") " Jul 14 16:07:46.287091 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:46.287057 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a-kube-api-access-dg79f" (OuterVolumeSpecName: "kube-api-access-dg79f") pod "e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a" (UID: "e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a"). InnerVolumeSpecName "kube-api-access-dg79f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 16:07:46.383059 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:46.383018 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7336-140624936488112"] Jul 14 16:07:46.383417 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:46.383403 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a" containerName="test-curl-7336-140624936488240" Jul 14 16:07:46.383467 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:46.383419 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a" containerName="test-curl-7336-140624936488240" Jul 14 16:07:46.383503 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:46.383488 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a" containerName="test-curl-7336-140624936488240" Jul 14 16:07:46.386021 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:46.386000 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg79f\" (UniqueName: \"kubernetes.io/projected/e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a-kube-api-access-dg79f\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:07:46.386534 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:46.386519 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936488112" Jul 14 16:07:46.393266 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:46.393238 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7336-140624936488112"] Jul 14 16:07:46.486450 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:46.486408 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-596k5\" (UniqueName: \"kubernetes.io/projected/d872d34d-5f3b-4470-b7ad-fdf22ed79b9e-kube-api-access-596k5\") pod \"test-curl-7336-140624936488112\" (UID: \"d872d34d-5f3b-4470-b7ad-fdf22ed79b9e\") " pod="openshift-ingress/test-curl-7336-140624936488112" Jul 14 16:07:46.587781 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:46.587741 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-596k5\" (UniqueName: \"kubernetes.io/projected/d872d34d-5f3b-4470-b7ad-fdf22ed79b9e-kube-api-access-596k5\") pod \"test-curl-7336-140624936488112\" (UID: \"d872d34d-5f3b-4470-b7ad-fdf22ed79b9e\") " pod="openshift-ingress/test-curl-7336-140624936488112" Jul 14 16:07:46.600094 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:46.600056 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-596k5\" (UniqueName: \"kubernetes.io/projected/d872d34d-5f3b-4470-b7ad-fdf22ed79b9e-kube-api-access-596k5\") pod \"test-curl-7336-140624936488112\" (UID: \"d872d34d-5f3b-4470-b7ad-fdf22ed79b9e\") " pod="openshift-ingress/test-curl-7336-140624936488112" Jul 14 16:07:46.697316 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:46.697227 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936488112" Jul 14 16:07:46.845396 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:46.845368 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7336-140624936488112"] Jul 14 16:07:46.847753 ip-10-0-128-22 kubenswrapper[2579]: W0714 16:07:46.847723 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd872d34d_5f3b_4470_b7ad_fdf22ed79b9e.slice/crio-e7fc82d6e0f99d2ed34d2392c53071eb84a1c64a5a0bdb9fd97c42dfd58fde0b WatchSource:0}: Error finding container e7fc82d6e0f99d2ed34d2392c53071eb84a1c64a5a0bdb9fd97c42dfd58fde0b: Status 404 returned error can't find the container with id e7fc82d6e0f99d2ed34d2392c53071eb84a1c64a5a0bdb9fd97c42dfd58fde0b Jul 14 16:07:47.051870 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:47.051774 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7336-140624936488112" event={"ID":"d872d34d-5f3b-4470-b7ad-fdf22ed79b9e","Type":"ContainerStarted","Data":"e7fc82d6e0f99d2ed34d2392c53071eb84a1c64a5a0bdb9fd97c42dfd58fde0b"} Jul 14 16:07:47.053016 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:47.052991 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbfb34d59ea54f1dff3d5df0ab40f9b4cb8328406d62b7f5fedc7113b89f4e13" Jul 14 16:07:47.053157 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:47.053015 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936488240" Jul 14 16:07:47.064237 ip-10-0-128-22 kubenswrapper[2579]: E0714 16:07:47.064208 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7336-140624936488240\" is forbidden: User \"system:node:ip-10-0-128-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-22.ec2.internal' and this object" podUID="e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a" pod="openshift-ingress/test-curl-7336-140624936488240" Jul 14 16:07:47.696603 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:47.696570 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a" path="/var/lib/kubelet/pods/e9449b85-a3dd-4f12-8d7f-728cbd0a3a3a/volumes" Jul 14 16:07:48.060478 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:48.060385 2579 generic.go:356] "Generic (PLEG): container finished" podID="d872d34d-5f3b-4470-b7ad-fdf22ed79b9e" containerID="5fe430bac71a0929a1164412f9dd697bf1c93fec5ef27877f8e6e8de6cbcb88b" exitCode=0 Jul 14 16:07:48.060657 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:48.060482 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7336-140624936488112" event={"ID":"d872d34d-5f3b-4470-b7ad-fdf22ed79b9e","Type":"ContainerDied","Data":"5fe430bac71a0929a1164412f9dd697bf1c93fec5ef27877f8e6e8de6cbcb88b"} Jul 14 16:07:49.199831 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:49.199807 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936488112" Jul 14 16:07:49.209499 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:49.209474 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-596k5\" (UniqueName: \"kubernetes.io/projected/d872d34d-5f3b-4470-b7ad-fdf22ed79b9e-kube-api-access-596k5\") pod \"d872d34d-5f3b-4470-b7ad-fdf22ed79b9e\" (UID: \"d872d34d-5f3b-4470-b7ad-fdf22ed79b9e\") " Jul 14 16:07:49.211754 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:49.211732 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d872d34d-5f3b-4470-b7ad-fdf22ed79b9e-kube-api-access-596k5" (OuterVolumeSpecName: "kube-api-access-596k5") pod "d872d34d-5f3b-4470-b7ad-fdf22ed79b9e" (UID: "d872d34d-5f3b-4470-b7ad-fdf22ed79b9e"). InnerVolumeSpecName "kube-api-access-596k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 16:07:49.226216 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:49.226186 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7336-140624936488112_d872d34d-5f3b-4470-b7ad-fdf22ed79b9e/test-curl-7336-140624936488112/0.log" Jul 14 16:07:49.258096 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:49.258043 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7336-140624936488112"] Jul 14 16:07:49.262697 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:49.262656 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7336-140624936488112"] Jul 14 16:07:49.310048 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:49.310014 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-596k5\" (UniqueName: \"kubernetes.io/projected/d872d34d-5f3b-4470-b7ad-fdf22ed79b9e-kube-api-access-596k5\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:07:49.435431 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:49.435393 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7336-140624936508848"] Jul 14 16:07:49.435797 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:49.435784 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d872d34d-5f3b-4470-b7ad-fdf22ed79b9e" containerName="test-curl-7336-140624936488112" Jul 14 16:07:49.435843 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:49.435799 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d872d34d-5f3b-4470-b7ad-fdf22ed79b9e" containerName="test-curl-7336-140624936488112" Jul 14 16:07:49.435878 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:49.435871 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d872d34d-5f3b-4470-b7ad-fdf22ed79b9e" containerName="test-curl-7336-140624936488112" Jul 14 16:07:49.438841 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:49.438821 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936508848" Jul 14 16:07:49.442763 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:49.442736 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7336-140624936508848"] Jul 14 16:07:49.511328 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:49.511291 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4745r\" (UniqueName: \"kubernetes.io/projected/bc6be08e-c560-46ea-84e8-cfab91bf6d22-kube-api-access-4745r\") pod \"test-curl-7336-140624936508848\" (UID: \"bc6be08e-c560-46ea-84e8-cfab91bf6d22\") " pod="openshift-ingress/test-curl-7336-140624936508848" Jul 14 16:07:49.612022 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:49.611982 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4745r\" (UniqueName: \"kubernetes.io/projected/bc6be08e-c560-46ea-84e8-cfab91bf6d22-kube-api-access-4745r\") pod \"test-curl-7336-140624936508848\" (UID: \"bc6be08e-c560-46ea-84e8-cfab91bf6d22\") " pod="openshift-ingress/test-curl-7336-140624936508848" Jul 14 16:07:49.622820 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:49.622787 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4745r\" (UniqueName: \"kubernetes.io/projected/bc6be08e-c560-46ea-84e8-cfab91bf6d22-kube-api-access-4745r\") pod \"test-curl-7336-140624936508848\" (UID: \"bc6be08e-c560-46ea-84e8-cfab91bf6d22\") " pod="openshift-ingress/test-curl-7336-140624936508848" Jul 14 16:07:49.698263 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:49.698158 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d872d34d-5f3b-4470-b7ad-fdf22ed79b9e" path="/var/lib/kubelet/pods/d872d34d-5f3b-4470-b7ad-fdf22ed79b9e/volumes" Jul 14 16:07:49.749443 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:49.749405 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936508848" Jul 14 16:07:49.889358 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:49.889315 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7336-140624936508848"] Jul 14 16:07:49.891271 ip-10-0-128-22 kubenswrapper[2579]: W0714 16:07:49.891235 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc6be08e_c560_46ea_84e8_cfab91bf6d22.slice/crio-8a6614bc615bcf87d341752e0410b248421d5551df04ecdd8f2770be1c4ab020 WatchSource:0}: Error finding container 8a6614bc615bcf87d341752e0410b248421d5551df04ecdd8f2770be1c4ab020: Status 404 returned error can't find the container with id 8a6614bc615bcf87d341752e0410b248421d5551df04ecdd8f2770be1c4ab020 Jul 14 16:07:50.072221 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:50.072159 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7336-140624936508848" event={"ID":"bc6be08e-c560-46ea-84e8-cfab91bf6d22","Type":"ContainerStarted","Data":"8a6614bc615bcf87d341752e0410b248421d5551df04ecdd8f2770be1c4ab020"} Jul 14 16:07:50.073409 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:50.073390 2579 scope.go:117] "RemoveContainer" containerID="5fe430bac71a0929a1164412f9dd697bf1c93fec5ef27877f8e6e8de6cbcb88b" Jul 14 16:07:50.073531 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:50.073391 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936488112" Jul 14 16:07:50.295256 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:50.295227 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:07:51.081919 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:51.081884 2579 generic.go:356] "Generic (PLEG): container finished" podID="bc6be08e-c560-46ea-84e8-cfab91bf6d22" containerID="d329729669debe4f9654815e2fb95db1d770d3aba29ab4a791767e1e3ccc3941" exitCode=0 Jul 14 16:07:51.082128 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:51.081964 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7336-140624936508848" event={"ID":"bc6be08e-c560-46ea-84e8-cfab91bf6d22","Type":"ContainerDied","Data":"d329729669debe4f9654815e2fb95db1d770d3aba29ab4a791767e1e3ccc3941"} Jul 14 16:07:52.222124 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:52.222094 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936508848" Jul 14 16:07:52.229448 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:52.229417 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4745r\" (UniqueName: \"kubernetes.io/projected/bc6be08e-c560-46ea-84e8-cfab91bf6d22-kube-api-access-4745r\") pod \"bc6be08e-c560-46ea-84e8-cfab91bf6d22\" (UID: \"bc6be08e-c560-46ea-84e8-cfab91bf6d22\") " Jul 14 16:07:52.231706 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:52.231681 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6be08e-c560-46ea-84e8-cfab91bf6d22-kube-api-access-4745r" (OuterVolumeSpecName: "kube-api-access-4745r") pod "bc6be08e-c560-46ea-84e8-cfab91bf6d22" (UID: "bc6be08e-c560-46ea-84e8-cfab91bf6d22"). InnerVolumeSpecName "kube-api-access-4745r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 16:07:52.263310 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:52.263255 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7336-140624936508848_bc6be08e-c560-46ea-84e8-cfab91bf6d22/test-curl-7336-140624936508848/0.log" Jul 14 16:07:52.306562 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:52.306519 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7336-140624936508848"] Jul 14 16:07:52.312497 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:52.312460 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7336-140624936508848"] Jul 14 16:07:52.330594 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:52.330561 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4745r\" (UniqueName: \"kubernetes.io/projected/bc6be08e-c560-46ea-84e8-cfab91bf6d22-kube-api-access-4745r\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:07:52.443208 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:52.443084 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7336-140624936508848"] Jul 14 16:07:52.443557 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:52.443537 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc6be08e-c560-46ea-84e8-cfab91bf6d22" containerName="test-curl-7336-140624936508848" Jul 14 16:07:52.443641 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:52.443559 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc6be08e-c560-46ea-84e8-cfab91bf6d22" containerName="test-curl-7336-140624936508848" Jul 14 16:07:52.443691 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:52.443664 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc6be08e-c560-46ea-84e8-cfab91bf6d22" containerName="test-curl-7336-140624936508848" Jul 14 16:07:52.446620 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:52.446596 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936508848" Jul 14 16:07:52.459877 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:52.459846 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7336-140624936508848"] Jul 14 16:07:52.531538 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:52.531500 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjs69\" (UniqueName: \"kubernetes.io/projected/c7b7e34e-c8f9-4b91-bd1f-35b656ee8837-kube-api-access-bjs69\") pod \"test-curl-7336-140624936508848\" (UID: \"c7b7e34e-c8f9-4b91-bd1f-35b656ee8837\") " pod="openshift-ingress/test-curl-7336-140624936508848" Jul 14 16:07:52.632016 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:52.631980 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjs69\" (UniqueName: \"kubernetes.io/projected/c7b7e34e-c8f9-4b91-bd1f-35b656ee8837-kube-api-access-bjs69\") pod \"test-curl-7336-140624936508848\" (UID: \"c7b7e34e-c8f9-4b91-bd1f-35b656ee8837\") " pod="openshift-ingress/test-curl-7336-140624936508848" Jul 14 16:07:52.644732 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:52.644697 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjs69\" (UniqueName: \"kubernetes.io/projected/c7b7e34e-c8f9-4b91-bd1f-35b656ee8837-kube-api-access-bjs69\") pod \"test-curl-7336-140624936508848\" (UID: \"c7b7e34e-c8f9-4b91-bd1f-35b656ee8837\") " pod="openshift-ingress/test-curl-7336-140624936508848" Jul 14 16:07:52.756020 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:52.755923 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936508848" Jul 14 16:07:52.896587 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:52.896553 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7336-140624936508848"] Jul 14 16:07:52.900439 ip-10-0-128-22 kubenswrapper[2579]: W0714 16:07:52.900408 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7b7e34e_c8f9_4b91_bd1f_35b656ee8837.slice/crio-3df36b6f1640e91e8f9930c1574f150a18798be2a28416e834b26a8e0d01b738 WatchSource:0}: Error finding container 3df36b6f1640e91e8f9930c1574f150a18798be2a28416e834b26a8e0d01b738: Status 404 returned error can't find the container with id 3df36b6f1640e91e8f9930c1574f150a18798be2a28416e834b26a8e0d01b738 Jul 14 16:07:53.094033 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:53.094001 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936508848" Jul 14 16:07:53.094033 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:53.094021 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a6614bc615bcf87d341752e0410b248421d5551df04ecdd8f2770be1c4ab020" Jul 14 16:07:53.095414 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:53.095385 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7336-140624936508848" event={"ID":"c7b7e34e-c8f9-4b91-bd1f-35b656ee8837","Type":"ContainerStarted","Data":"3df36b6f1640e91e8f9930c1574f150a18798be2a28416e834b26a8e0d01b738"} Jul 14 16:07:53.106735 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:53.106703 2579 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7336-140624936508848" oldPodUID="bc6be08e-c560-46ea-84e8-cfab91bf6d22" podUID="c7b7e34e-c8f9-4b91-bd1f-35b656ee8837" Jul 14 16:07:53.697236 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:53.697199 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc6be08e-c560-46ea-84e8-cfab91bf6d22" path="/var/lib/kubelet/pods/bc6be08e-c560-46ea-84e8-cfab91bf6d22/volumes" Jul 14 16:07:54.102008 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:54.101965 2579 generic.go:356] "Generic (PLEG): container finished" podID="c7b7e34e-c8f9-4b91-bd1f-35b656ee8837" containerID="db9d889766edf2bbe492a395450fdf93c3c8b7530f2cdc73a3140fa3f70bb232" exitCode=0 Jul 14 16:07:54.102223 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:54.102050 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7336-140624936508848" event={"ID":"c7b7e34e-c8f9-4b91-bd1f-35b656ee8837","Type":"ContainerDied","Data":"db9d889766edf2bbe492a395450fdf93c3c8b7530f2cdc73a3140fa3f70bb232"} Jul 14 16:07:55.248414 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:55.248373 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936508848" Jul 14 16:07:55.252923 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:55.252901 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjs69\" (UniqueName: \"kubernetes.io/projected/c7b7e34e-c8f9-4b91-bd1f-35b656ee8837-kube-api-access-bjs69\") pod \"c7b7e34e-c8f9-4b91-bd1f-35b656ee8837\" (UID: \"c7b7e34e-c8f9-4b91-bd1f-35b656ee8837\") " Jul 14 16:07:55.255134 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:55.255110 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b7e34e-c8f9-4b91-bd1f-35b656ee8837-kube-api-access-bjs69" (OuterVolumeSpecName: "kube-api-access-bjs69") pod "c7b7e34e-c8f9-4b91-bd1f-35b656ee8837" (UID: "c7b7e34e-c8f9-4b91-bd1f-35b656ee8837"). InnerVolumeSpecName "kube-api-access-bjs69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 16:07:55.270991 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:55.270967 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7336-140624936508848_c7b7e34e-c8f9-4b91-bd1f-35b656ee8837/test-curl-7336-140624936508848/0.log" Jul 14 16:07:55.302912 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:55.302882 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7336-140624936508848"] Jul 14 16:07:55.304986 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:55.304965 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7336-140624936508848"] Jul 14 16:07:55.354015 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:55.353974 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjs69\" (UniqueName: \"kubernetes.io/projected/c7b7e34e-c8f9-4b91-bd1f-35b656ee8837-kube-api-access-bjs69\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:07:55.448379 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:55.448290 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7336-140624936460768"] Jul 14 16:07:55.448701 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:55.448686 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7b7e34e-c8f9-4b91-bd1f-35b656ee8837" containerName="test-curl-7336-140624936508848" Jul 14 16:07:55.448745 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:55.448704 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7b7e34e-c8f9-4b91-bd1f-35b656ee8837" containerName="test-curl-7336-140624936508848" Jul 14 16:07:55.448784 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:55.448773 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7b7e34e-c8f9-4b91-bd1f-35b656ee8837" containerName="test-curl-7336-140624936508848" Jul 14 16:07:55.451897 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:55.451881 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936460768" Jul 14 16:07:55.454853 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:55.454826 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqvzt\" (UniqueName: \"kubernetes.io/projected/c58929bb-f6cb-4751-ade8-7378f4199ea7-kube-api-access-dqvzt\") pod \"test-curl-7336-140624936460768\" (UID: \"c58929bb-f6cb-4751-ade8-7378f4199ea7\") " pod="openshift-ingress/test-curl-7336-140624936460768" Jul 14 16:07:55.457518 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:55.457493 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7336-140624936460768"] Jul 14 16:07:55.555335 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:55.555297 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqvzt\" (UniqueName: \"kubernetes.io/projected/c58929bb-f6cb-4751-ade8-7378f4199ea7-kube-api-access-dqvzt\") pod \"test-curl-7336-140624936460768\" (UID: \"c58929bb-f6cb-4751-ade8-7378f4199ea7\") " pod="openshift-ingress/test-curl-7336-140624936460768" Jul 14 16:07:55.565242 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:55.565208 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqvzt\" (UniqueName: \"kubernetes.io/projected/c58929bb-f6cb-4751-ade8-7378f4199ea7-kube-api-access-dqvzt\") pod \"test-curl-7336-140624936460768\" (UID: \"c58929bb-f6cb-4751-ade8-7378f4199ea7\") " pod="openshift-ingress/test-curl-7336-140624936460768" Jul 14 16:07:55.697022 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:55.696985 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7b7e34e-c8f9-4b91-bd1f-35b656ee8837" path="/var/lib/kubelet/pods/c7b7e34e-c8f9-4b91-bd1f-35b656ee8837/volumes" Jul 14 16:07:55.762643 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:55.762598 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936460768" Jul 14 16:07:55.904992 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:55.904963 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7336-140624936460768"] Jul 14 16:07:55.906662 ip-10-0-128-22 kubenswrapper[2579]: W0714 16:07:55.906635 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc58929bb_f6cb_4751_ade8_7378f4199ea7.slice/crio-ba36aa69dae1ec679bd9a564fa7c80db5e729234e57f43c4e00c878f75a9c379 WatchSource:0}: Error finding container ba36aa69dae1ec679bd9a564fa7c80db5e729234e57f43c4e00c878f75a9c379: Status 404 returned error can't find the container with id ba36aa69dae1ec679bd9a564fa7c80db5e729234e57f43c4e00c878f75a9c379 Jul 14 16:07:56.115521 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.115484 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7336-140624936460768" event={"ID":"c58929bb-f6cb-4751-ade8-7378f4199ea7","Type":"ContainerStarted","Data":"ba36aa69dae1ec679bd9a564fa7c80db5e729234e57f43c4e00c878f75a9c379"} Jul 14 16:07:56.116680 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.116661 2579 scope.go:117] "RemoveContainer" containerID="db9d889766edf2bbe492a395450fdf93c3c8b7530f2cdc73a3140fa3f70bb232" Jul 14 16:07:56.116803 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.116663 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936508848" Jul 14 16:07:56.300611 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.300570 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" podUID="f04682d2-df47-4e81-a179-a9c359815e1e" containerName="main" containerID="cri-o://16e30b8ba7cdc40c89777f7607e246e1239a47781dd329b39d9dbb1728925cae" gracePeriod=45 Jul 14 16:07:56.554673 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.554645 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:07:56.564991 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.564962 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4kd5\" (UniqueName: \"kubernetes.io/projected/f04682d2-df47-4e81-a179-a9c359815e1e-kube-api-access-g4kd5\") pod \"f04682d2-df47-4e81-a179-a9c359815e1e\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " Jul 14 16:07:56.565101 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.564996 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-home\") pod \"f04682d2-df47-4e81-a179-a9c359815e1e\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " Jul 14 16:07:56.565101 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.565025 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-dshm\") pod \"f04682d2-df47-4e81-a179-a9c359815e1e\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " Jul 14 16:07:56.565101 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.565044 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f04682d2-df47-4e81-a179-a9c359815e1e-tls-certs\") pod \"f04682d2-df47-4e81-a179-a9c359815e1e\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " Jul 14 16:07:56.565101 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.565096 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-tmp-dir\") pod \"f04682d2-df47-4e81-a179-a9c359815e1e\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " Jul 14 16:07:56.565315 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.565125 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-model-cache\") pod \"f04682d2-df47-4e81-a179-a9c359815e1e\" (UID: \"f04682d2-df47-4e81-a179-a9c359815e1e\") " Jul 14 16:07:56.565375 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.565315 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-home" (OuterVolumeSpecName: "home") pod "f04682d2-df47-4e81-a179-a9c359815e1e" (UID: "f04682d2-df47-4e81-a179-a9c359815e1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:07:56.565462 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.565432 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f04682d2-df47-4e81-a179-a9c359815e1e" (UID: "f04682d2-df47-4e81-a179-a9c359815e1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:07:56.565462 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.565455 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-home\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:07:56.565547 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.565448 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f04682d2-df47-4e81-a179-a9c359815e1e" (UID: "f04682d2-df47-4e81-a179-a9c359815e1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:07:56.567480 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.567458 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f04682d2-df47-4e81-a179-a9c359815e1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f04682d2-df47-4e81-a179-a9c359815e1e" (UID: "f04682d2-df47-4e81-a179-a9c359815e1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 16:07:56.567755 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.567736 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04682d2-df47-4e81-a179-a9c359815e1e-kube-api-access-g4kd5" (OuterVolumeSpecName: "kube-api-access-g4kd5") pod "f04682d2-df47-4e81-a179-a9c359815e1e" (UID: "f04682d2-df47-4e81-a179-a9c359815e1e"). InnerVolumeSpecName "kube-api-access-g4kd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 16:07:56.567755 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.567743 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-dshm" (OuterVolumeSpecName: "dshm") pod "f04682d2-df47-4e81-a179-a9c359815e1e" (UID: "f04682d2-df47-4e81-a179-a9c359815e1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:07:56.666147 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.666046 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-tmp-dir\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:07:56.666147 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.666091 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-model-cache\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:07:56.666147 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.666104 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4kd5\" (UniqueName: \"kubernetes.io/projected/f04682d2-df47-4e81-a179-a9c359815e1e-kube-api-access-g4kd5\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:07:56.666147 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.666113 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f04682d2-df47-4e81-a179-a9c359815e1e-dshm\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:07:56.666147 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:56.666122 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f04682d2-df47-4e81-a179-a9c359815e1e-tls-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:07:57.124049 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.124011 2579 generic.go:356] "Generic (PLEG): container finished" podID="c58929bb-f6cb-4751-ade8-7378f4199ea7" containerID="d67e41aa39ea501e96d96f63eb93edc7707b7c1107bdb6e006faa5602c766f0d" exitCode=0 Jul 14 16:07:57.124288 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.124101 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7336-140624936460768" event={"ID":"c58929bb-f6cb-4751-ade8-7378f4199ea7","Type":"ContainerDied","Data":"d67e41aa39ea501e96d96f63eb93edc7707b7c1107bdb6e006faa5602c766f0d"} Jul 14 16:07:57.125489 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.125464 2579 generic.go:356] "Generic (PLEG): container finished" podID="f04682d2-df47-4e81-a179-a9c359815e1e" containerID="16e30b8ba7cdc40c89777f7607e246e1239a47781dd329b39d9dbb1728925cae" exitCode=0 Jul 14 16:07:57.125622 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.125500 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" Jul 14 16:07:57.125622 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.125560 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" event={"ID":"f04682d2-df47-4e81-a179-a9c359815e1e","Type":"ContainerDied","Data":"16e30b8ba7cdc40c89777f7607e246e1239a47781dd329b39d9dbb1728925cae"} Jul 14 16:07:57.125622 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.125611 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg" event={"ID":"f04682d2-df47-4e81-a179-a9c359815e1e","Type":"ContainerDied","Data":"16ca995876da8e3aea59a77450e43ae8e128db8d1426f2e5eaeb692f3eb9b1d2"} Jul 14 16:07:57.125754 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.125637 2579 scope.go:117] "RemoveContainer" containerID="16e30b8ba7cdc40c89777f7607e246e1239a47781dd329b39d9dbb1728925cae" Jul 14 16:07:57.136858 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.136842 2579 scope.go:117] "RemoveContainer" containerID="16e30b8ba7cdc40c89777f7607e246e1239a47781dd329b39d9dbb1728925cae" Jul 14 16:07:57.137128 ip-10-0-128-22 kubenswrapper[2579]: E0714 16:07:57.137107 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16e30b8ba7cdc40c89777f7607e246e1239a47781dd329b39d9dbb1728925cae\": container with ID starting with 16e30b8ba7cdc40c89777f7607e246e1239a47781dd329b39d9dbb1728925cae not found: ID does not exist" containerID="16e30b8ba7cdc40c89777f7607e246e1239a47781dd329b39d9dbb1728925cae" Jul 14 16:07:57.137228 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.137137 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16e30b8ba7cdc40c89777f7607e246e1239a47781dd329b39d9dbb1728925cae"} err="failed to get container status \"16e30b8ba7cdc40c89777f7607e246e1239a47781dd329b39d9dbb1728925cae\": rpc error: code = NotFound desc = could not find container \"16e30b8ba7cdc40c89777f7607e246e1239a47781dd329b39d9dbb1728925cae\": container with ID starting with 16e30b8ba7cdc40c89777f7607e246e1239a47781dd329b39d9dbb1728925cae not found: ID does not exist" Jul 14 16:07:57.159298 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.159266 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg"] Jul 14 16:07:57.165724 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.165698 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bea800/rate-test-model-bea800-kserve-66df66fcd7-vrntg"] Jul 14 16:07:57.238074 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.238016 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" podUID="25c848bb-84cd-4f57-b2d0-1aaf1c54ecac" containerName="main" containerID="cri-o://a499f5b08fbbaea5f5d8a3f5d8750593bb99c96ffc79f11c7bc015a961fd543d" gracePeriod=45 Jul 14 16:07:57.505643 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.505612 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:07:57.572994 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.572956 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-tls-certs\") pod \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " Jul 14 16:07:57.573205 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.573013 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-model-cache\") pod \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " Jul 14 16:07:57.573205 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.573086 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckrw4\" (UniqueName: \"kubernetes.io/projected/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-kube-api-access-ckrw4\") pod \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " Jul 14 16:07:57.573205 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.573121 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-home\") pod \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " Jul 14 16:07:57.573205 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.573162 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-dshm\") pod \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " Jul 14 16:07:57.573430 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.573223 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-tmp-dir\") pod \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\" (UID: \"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac\") " Jul 14 16:07:57.573430 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.573343 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-model-cache" (OuterVolumeSpecName: "model-cache") pod "25c848bb-84cd-4f57-b2d0-1aaf1c54ecac" (UID: "25c848bb-84cd-4f57-b2d0-1aaf1c54ecac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:07:57.573530 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.573455 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-model-cache\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:07:57.573530 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.573466 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-home" (OuterVolumeSpecName: "home") pod "25c848bb-84cd-4f57-b2d0-1aaf1c54ecac" (UID: "25c848bb-84cd-4f57-b2d0-1aaf1c54ecac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:07:57.573530 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.573506 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "25c848bb-84cd-4f57-b2d0-1aaf1c54ecac" (UID: "25c848bb-84cd-4f57-b2d0-1aaf1c54ecac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:07:57.575426 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.575399 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-dshm" (OuterVolumeSpecName: "dshm") pod "25c848bb-84cd-4f57-b2d0-1aaf1c54ecac" (UID: "25c848bb-84cd-4f57-b2d0-1aaf1c54ecac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 16:07:57.575544 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.575468 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-kube-api-access-ckrw4" (OuterVolumeSpecName: "kube-api-access-ckrw4") pod "25c848bb-84cd-4f57-b2d0-1aaf1c54ecac" (UID: "25c848bb-84cd-4f57-b2d0-1aaf1c54ecac"). InnerVolumeSpecName "kube-api-access-ckrw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 16:07:57.575544 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.575494 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25c848bb-84cd-4f57-b2d0-1aaf1c54ecac" (UID: "25c848bb-84cd-4f57-b2d0-1aaf1c54ecac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 16:07:57.674698 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.674597 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-tls-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:07:57.674698 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.674636 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckrw4\" (UniqueName: \"kubernetes.io/projected/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-kube-api-access-ckrw4\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:07:57.674698 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.674648 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-home\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:07:57.674698 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.674656 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-dshm\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:07:57.674698 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.674664 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac-tmp-dir\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:07:57.700269 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:57.700231 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f04682d2-df47-4e81-a179-a9c359815e1e" path="/var/lib/kubelet/pods/f04682d2-df47-4e81-a179-a9c359815e1e/volumes" Jul 14 16:07:58.138382 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.138330 2579 generic.go:356] "Generic (PLEG): container finished" podID="25c848bb-84cd-4f57-b2d0-1aaf1c54ecac" containerID="a499f5b08fbbaea5f5d8a3f5d8750593bb99c96ffc79f11c7bc015a961fd543d" exitCode=0 Jul 14 16:07:58.138567 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.138419 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" Jul 14 16:07:58.138567 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.138420 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" event={"ID":"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac","Type":"ContainerDied","Data":"a499f5b08fbbaea5f5d8a3f5d8750593bb99c96ffc79f11c7bc015a961fd543d"} Jul 14 16:07:58.138567 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.138474 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2" event={"ID":"25c848bb-84cd-4f57-b2d0-1aaf1c54ecac","Type":"ContainerDied","Data":"540e8a2471ab3ae6a2da1dcc3ff967b0c8609d922f72dcdc3e3ca78699797bf0"} Jul 14 16:07:58.138567 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.138495 2579 scope.go:117] "RemoveContainer" containerID="a499f5b08fbbaea5f5d8a3f5d8750593bb99c96ffc79f11c7bc015a961fd543d" Jul 14 16:07:58.148859 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.148833 2579 scope.go:117] "RemoveContainer" containerID="a499f5b08fbbaea5f5d8a3f5d8750593bb99c96ffc79f11c7bc015a961fd543d" Jul 14 16:07:58.149207 ip-10-0-128-22 kubenswrapper[2579]: E0714 16:07:58.149159 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a499f5b08fbbaea5f5d8a3f5d8750593bb99c96ffc79f11c7bc015a961fd543d\": container with ID starting with a499f5b08fbbaea5f5d8a3f5d8750593bb99c96ffc79f11c7bc015a961fd543d not found: ID does not exist" containerID="a499f5b08fbbaea5f5d8a3f5d8750593bb99c96ffc79f11c7bc015a961fd543d" Jul 14 16:07:58.149315 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.149214 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a499f5b08fbbaea5f5d8a3f5d8750593bb99c96ffc79f11c7bc015a961fd543d"} err="failed to get container status \"a499f5b08fbbaea5f5d8a3f5d8750593bb99c96ffc79f11c7bc015a961fd543d\": rpc error: code = NotFound desc = could not find container \"a499f5b08fbbaea5f5d8a3f5d8750593bb99c96ffc79f11c7bc015a961fd543d\": container with ID starting with a499f5b08fbbaea5f5d8a3f5d8750593bb99c96ffc79f11c7bc015a961fd543d not found: ID does not exist" Jul 14 16:07:58.162526 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.162493 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2"] Jul 14 16:07:58.176297 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.176266 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-519447/rate-test-model-519447-kserve-5c9fd96c9f-cllv2"] Jul 14 16:07:58.274545 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.274518 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936460768" Jul 14 16:07:58.279503 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.279482 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqvzt\" (UniqueName: \"kubernetes.io/projected/c58929bb-f6cb-4751-ade8-7378f4199ea7-kube-api-access-dqvzt\") pod \"c58929bb-f6cb-4751-ade8-7378f4199ea7\" (UID: \"c58929bb-f6cb-4751-ade8-7378f4199ea7\") " Jul 14 16:07:58.281683 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.281664 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58929bb-f6cb-4751-ade8-7378f4199ea7-kube-api-access-dqvzt" (OuterVolumeSpecName: "kube-api-access-dqvzt") pod "c58929bb-f6cb-4751-ade8-7378f4199ea7" (UID: "c58929bb-f6cb-4751-ade8-7378f4199ea7"). InnerVolumeSpecName "kube-api-access-dqvzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 16:07:58.299342 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.299310 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7336-140624936460768_c58929bb-f6cb-4751-ade8-7378f4199ea7/test-curl-7336-140624936460768/0.log" Jul 14 16:07:58.328556 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.328527 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7336-140624936460768"] Jul 14 16:07:58.330618 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.330596 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7336-140624936460768"] Jul 14 16:07:58.380976 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.380935 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqvzt\" (UniqueName: \"kubernetes.io/projected/c58929bb-f6cb-4751-ade8-7378f4199ea7-kube-api-access-dqvzt\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:07:58.460109 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.460010 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7336-140624936460768"] Jul 14 16:07:58.460470 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.460451 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c58929bb-f6cb-4751-ade8-7378f4199ea7" containerName="test-curl-7336-140624936460768" Jul 14 16:07:58.460557 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.460474 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c58929bb-f6cb-4751-ade8-7378f4199ea7" containerName="test-curl-7336-140624936460768" Jul 14 16:07:58.460557 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.460488 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25c848bb-84cd-4f57-b2d0-1aaf1c54ecac" containerName="main" Jul 14 16:07:58.460557 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.460496 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="25c848bb-84cd-4f57-b2d0-1aaf1c54ecac" containerName="main" Jul 14 16:07:58.460557 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.460506 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f04682d2-df47-4e81-a179-a9c359815e1e" containerName="main" Jul 14 16:07:58.460557 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.460514 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f04682d2-df47-4e81-a179-a9c359815e1e" containerName="main" Jul 14 16:07:58.460787 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.460638 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f04682d2-df47-4e81-a179-a9c359815e1e" containerName="main" Jul 14 16:07:58.460787 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.460653 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="25c848bb-84cd-4f57-b2d0-1aaf1c54ecac" containerName="main" Jul 14 16:07:58.460787 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.460665 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c58929bb-f6cb-4751-ade8-7378f4199ea7" containerName="test-curl-7336-140624936460768" Jul 14 16:07:58.463760 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.463738 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936460768" Jul 14 16:07:58.469883 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.469859 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7336-140624936460768"] Jul 14 16:07:58.481966 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.481933 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2px4\" (UniqueName: \"kubernetes.io/projected/fb7e9e4f-2d8f-426f-b062-482b320bb523-kube-api-access-x2px4\") pod \"test-curl-7336-140624936460768\" (UID: \"fb7e9e4f-2d8f-426f-b062-482b320bb523\") " pod="openshift-ingress/test-curl-7336-140624936460768" Jul 14 16:07:58.583274 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.583233 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2px4\" (UniqueName: \"kubernetes.io/projected/fb7e9e4f-2d8f-426f-b062-482b320bb523-kube-api-access-x2px4\") pod \"test-curl-7336-140624936460768\" (UID: \"fb7e9e4f-2d8f-426f-b062-482b320bb523\") " pod="openshift-ingress/test-curl-7336-140624936460768" Jul 14 16:07:58.593025 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.592997 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2px4\" (UniqueName: \"kubernetes.io/projected/fb7e9e4f-2d8f-426f-b062-482b320bb523-kube-api-access-x2px4\") pod \"test-curl-7336-140624936460768\" (UID: \"fb7e9e4f-2d8f-426f-b062-482b320bb523\") " pod="openshift-ingress/test-curl-7336-140624936460768" Jul 14 16:07:58.774361 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.774253 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936460768" Jul 14 16:07:58.919579 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:58.919549 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7336-140624936460768"] Jul 14 16:07:58.921691 ip-10-0-128-22 kubenswrapper[2579]: W0714 16:07:58.921661 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb7e9e4f_2d8f_426f_b062_482b320bb523.slice/crio-4b0d8beac850e3ee5193f776d12471d2a5f99c413c14890ecdf7750887124c1a WatchSource:0}: Error finding container 4b0d8beac850e3ee5193f776d12471d2a5f99c413c14890ecdf7750887124c1a: Status 404 returned error can't find the container with id 4b0d8beac850e3ee5193f776d12471d2a5f99c413c14890ecdf7750887124c1a Jul 14 16:07:59.147351 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:59.147322 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936460768" Jul 14 16:07:59.147351 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:59.147342 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba36aa69dae1ec679bd9a564fa7c80db5e729234e57f43c4e00c878f75a9c379" Jul 14 16:07:59.148706 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:59.148680 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7336-140624936460768" event={"ID":"fb7e9e4f-2d8f-426f-b062-482b320bb523","Type":"ContainerStarted","Data":"4b0d8beac850e3ee5193f776d12471d2a5f99c413c14890ecdf7750887124c1a"} Jul 14 16:07:59.160351 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:59.160322 2579 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7336-140624936460768" oldPodUID="c58929bb-f6cb-4751-ade8-7378f4199ea7" podUID="fb7e9e4f-2d8f-426f-b062-482b320bb523" Jul 14 16:07:59.697613 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:59.697578 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25c848bb-84cd-4f57-b2d0-1aaf1c54ecac" path="/var/lib/kubelet/pods/25c848bb-84cd-4f57-b2d0-1aaf1c54ecac/volumes" Jul 14 16:07:59.697967 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:07:59.697898 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c58929bb-f6cb-4751-ade8-7378f4199ea7" path="/var/lib/kubelet/pods/c58929bb-f6cb-4751-ade8-7378f4199ea7/volumes" Jul 14 16:08:00.156718 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:00.156680 2579 generic.go:356] "Generic (PLEG): container finished" podID="fb7e9e4f-2d8f-426f-b062-482b320bb523" containerID="e4596d0b65ad82a63f84f8f973680fe107abdc808d5c99bf77eb574bc3531bcc" exitCode=0 Jul 14 16:08:00.156909 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:00.156721 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7336-140624936460768" event={"ID":"fb7e9e4f-2d8f-426f-b062-482b320bb523","Type":"ContainerDied","Data":"e4596d0b65ad82a63f84f8f973680fe107abdc808d5c99bf77eb574bc3531bcc"} Jul 14 16:08:01.296143 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:01.296113 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936460768" Jul 14 16:08:01.306635 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:01.306556 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2px4\" (UniqueName: \"kubernetes.io/projected/fb7e9e4f-2d8f-426f-b062-482b320bb523-kube-api-access-x2px4\") pod \"fb7e9e4f-2d8f-426f-b062-482b320bb523\" (UID: \"fb7e9e4f-2d8f-426f-b062-482b320bb523\") " Jul 14 16:08:01.308944 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:01.308915 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7e9e4f-2d8f-426f-b062-482b320bb523-kube-api-access-x2px4" (OuterVolumeSpecName: "kube-api-access-x2px4") pod "fb7e9e4f-2d8f-426f-b062-482b320bb523" (UID: "fb7e9e4f-2d8f-426f-b062-482b320bb523"). InnerVolumeSpecName "kube-api-access-x2px4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 16:08:01.320057 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:01.320029 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7336-140624936460768_fb7e9e4f-2d8f-426f-b062-482b320bb523/test-curl-7336-140624936460768/0.log" Jul 14 16:08:01.348478 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:01.348443 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7336-140624936460768"] Jul 14 16:08:01.352802 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:01.352772 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7336-140624936460768"] Jul 14 16:08:01.407935 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:01.407895 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2px4\" (UniqueName: \"kubernetes.io/projected/fb7e9e4f-2d8f-426f-b062-482b320bb523-kube-api-access-x2px4\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:08:01.551365 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:01.551324 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-bea800-pwd2v"] Jul 14 16:08:01.551706 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:01.551693 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb7e9e4f-2d8f-426f-b062-482b320bb523" containerName="test-curl-7336-140624936460768" Jul 14 16:08:01.551745 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:01.551707 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb7e9e4f-2d8f-426f-b062-482b320bb523" containerName="test-curl-7336-140624936460768" Jul 14 16:08:01.551776 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:01.551764 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb7e9e4f-2d8f-426f-b062-482b320bb523" containerName="test-curl-7336-140624936460768" Jul 14 16:08:01.554669 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:01.554652 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-bea800-pwd2v" Jul 14 16:08:01.557628 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:01.557550 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4dcsv\"" Jul 14 16:08:01.557764 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:01.557662 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 16:08:01.562890 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:01.562868 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-bea800-pwd2v"] Jul 14 16:08:01.609868 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:01.609826 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a6572d-12ed-41a0-b1dc-be196e8f03dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-bea800-pwd2v\" (UID: \"62a6572d-12ed-41a0-b1dc-be196e8f03dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-bea800-pwd2v" Jul 14 16:08:01.703043 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:01.703009 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb7e9e4f-2d8f-426f-b062-482b320bb523" path="/var/lib/kubelet/pods/fb7e9e4f-2d8f-426f-b062-482b320bb523/volumes" Jul 14 16:08:01.710351 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:01.710312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a6572d-12ed-41a0-b1dc-be196e8f03dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-bea800-pwd2v\" (UID: \"62a6572d-12ed-41a0-b1dc-be196e8f03dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-bea800-pwd2v" Jul 14 16:08:01.711139 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:01.711114 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a6572d-12ed-41a0-b1dc-be196e8f03dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-bea800-pwd2v\" (UID: \"62a6572d-12ed-41a0-b1dc-be196e8f03dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-bea800-pwd2v" Jul 14 16:08:01.866890 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:01.866856 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-bea800-pwd2v" Jul 14 16:08:02.009593 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:02.009564 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-bea800-pwd2v"] Jul 14 16:08:02.012069 ip-10-0-128-22 kubenswrapper[2579]: W0714 16:08:02.012039 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a6572d_12ed_41a0_b1dc_be196e8f03dc.slice/crio-a545f8884077abc6456e2a475bfacf71d2c47f1cadde5c5a899f27f10c52194a WatchSource:0}: Error finding container a545f8884077abc6456e2a475bfacf71d2c47f1cadde5c5a899f27f10c52194a: Status 404 returned error can't find the container with id a545f8884077abc6456e2a475bfacf71d2c47f1cadde5c5a899f27f10c52194a Jul 14 16:08:02.112719 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:02.112671 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:02.169845 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:02.169740 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-bea800-pwd2v" event={"ID":"62a6572d-12ed-41a0-b1dc-be196e8f03dc","Type":"ContainerStarted","Data":"a545f8884077abc6456e2a475bfacf71d2c47f1cadde5c5a899f27f10c52194a"} Jul 14 16:08:02.171165 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:02.171133 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7336-140624936460768" Jul 14 16:08:02.171347 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:02.171146 2579 scope.go:117] "RemoveContainer" containerID="e4596d0b65ad82a63f84f8f973680fe107abdc808d5c99bf77eb574bc3531bcc" Jul 14 16:08:02.336550 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:02.336511 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:03.178966 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:03.178929 2579 generic.go:356] "Generic (PLEG): container finished" podID="62a6572d-12ed-41a0-b1dc-be196e8f03dc" containerID="35627123cad6d03161755552ab9823570640a86c25aaec3d8917a5a2ee998829" exitCode=0 Jul 14 16:08:03.179119 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:03.178990 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-bea800-pwd2v" event={"ID":"62a6572d-12ed-41a0-b1dc-be196e8f03dc","Type":"ContainerDied","Data":"35627123cad6d03161755552ab9823570640a86c25aaec3d8917a5a2ee998829"} Jul 14 16:08:04.343529 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:04.343506 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-bea800-pwd2v" Jul 14 16:08:04.432271 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:04.432229 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a6572d-12ed-41a0-b1dc-be196e8f03dc-maas-api-service-ca\") pod \"62a6572d-12ed-41a0-b1dc-be196e8f03dc\" (UID: \"62a6572d-12ed-41a0-b1dc-be196e8f03dc\") " Jul 14 16:08:04.432597 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:04.432571 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a6572d-12ed-41a0-b1dc-be196e8f03dc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62a6572d-12ed-41a0-b1dc-be196e8f03dc" (UID: "62a6572d-12ed-41a0-b1dc-be196e8f03dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 16:08:04.533579 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:04.533485 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a6572d-12ed-41a0-b1dc-be196e8f03dc-maas-api-service-ca\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:08:05.191935 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:05.191901 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-bea800-pwd2v" Jul 14 16:08:05.191935 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:05.191930 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-bea800-pwd2v" event={"ID":"62a6572d-12ed-41a0-b1dc-be196e8f03dc","Type":"ContainerDied","Data":"a545f8884077abc6456e2a475bfacf71d2c47f1cadde5c5a899f27f10c52194a"} Jul 14 16:08:05.192192 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:05.191976 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a545f8884077abc6456e2a475bfacf71d2c47f1cadde5c5a899f27f10c52194a" Jul 14 16:08:12.124792 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:12.124752 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:13.763500 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:13.763462 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:15.595287 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:15.595248 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:08:16.673440 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:16.673406 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:23.133532 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:23.133500 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:23.198596 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:23.198557 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:27.433537 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:27.433486 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:27.685445 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:27.685358 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:27.813430 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:27.813382 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-519447-5sfj4"] Jul 14 16:08:27.813821 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:27.813802 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a6572d-12ed-41a0-b1dc-be196e8f03dc" containerName="revoke-keys" Jul 14 16:08:27.813915 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:27.813824 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a6572d-12ed-41a0-b1dc-be196e8f03dc" containerName="revoke-keys" Jul 14 16:08:27.813971 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:27.813919 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a6572d-12ed-41a0-b1dc-be196e8f03dc" containerName="revoke-keys" Jul 14 16:08:27.818579 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:27.818554 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-519447-5sfj4" Jul 14 16:08:27.821942 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:27.821608 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 16:08:27.821942 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:27.821614 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4dcsv\"" Jul 14 16:08:27.826248 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:27.826224 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-519447-5sfj4"] Jul 14 16:08:27.938294 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:27.938189 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-519447-5sfj4\" (UID: \"9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-519447-5sfj4" Jul 14 16:08:28.039852 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:28.039809 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-519447-5sfj4\" (UID: \"9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-519447-5sfj4" Jul 14 16:08:28.040466 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:28.040448 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-519447-5sfj4\" (UID: \"9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-519447-5sfj4" Jul 14 16:08:28.130832 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:28.130788 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-519447-5sfj4" Jul 14 16:08:28.274694 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:28.274669 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-519447-5sfj4"] Jul 14 16:08:28.276808 ip-10-0-128-22 kubenswrapper[2579]: W0714 16:08:28.276769 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f3a7e85_2fa4_4b98_bc42_bb54f4c7ce1b.slice/crio-b402e95f5c56f8acb04dd58dfcfa7bd53c21d73c00e4e4ba5cf4d797ce623906 WatchSource:0}: Error finding container b402e95f5c56f8acb04dd58dfcfa7bd53c21d73c00e4e4ba5cf4d797ce623906: Status 404 returned error can't find the container with id b402e95f5c56f8acb04dd58dfcfa7bd53c21d73c00e4e4ba5cf4d797ce623906 Jul 14 16:08:28.314474 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:28.314441 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-519447-5sfj4" event={"ID":"9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b","Type":"ContainerStarted","Data":"b402e95f5c56f8acb04dd58dfcfa7bd53c21d73c00e4e4ba5cf4d797ce623906"} Jul 14 16:08:28.394450 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:28.394397 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:28.494379 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:28.494333 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:29.321976 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:29.321940 2579 generic.go:356] "Generic (PLEG): container finished" podID="9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b" containerID="3ff5f5d83500ee3e1fe53c203adfb98c6e39c87cc2c8ec01588a3e50423945c1" exitCode=0 Jul 14 16:08:29.322214 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:29.322022 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-519447-5sfj4" event={"ID":"9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b","Type":"ContainerDied","Data":"3ff5f5d83500ee3e1fe53c203adfb98c6e39c87cc2c8ec01588a3e50423945c1"} Jul 14 16:08:30.465810 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:30.465781 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-519447-5sfj4" Jul 14 16:08:30.562512 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:30.562480 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b-maas-api-service-ca\") pod \"9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b\" (UID: \"9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b\") " Jul 14 16:08:30.562828 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:30.562804 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b" (UID: "9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 16:08:30.663122 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:30.663036 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b-maas-api-service-ca\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 14 16:08:31.334002 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:31.333957 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-519447-5sfj4" Jul 14 16:08:31.334002 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:31.333985 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-519447-5sfj4" event={"ID":"9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b","Type":"ContainerDied","Data":"b402e95f5c56f8acb04dd58dfcfa7bd53c21d73c00e4e4ba5cf4d797ce623906"} Jul 14 16:08:31.334265 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:31.334044 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b402e95f5c56f8acb04dd58dfcfa7bd53c21d73c00e4e4ba5cf4d797ce623906" Jul 14 16:08:34.142301 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:34.142262 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:38.398516 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:38.398475 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:41.704090 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:41.704049 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-28mc5"] Jul 14 16:08:42.920456 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:42.920422 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:47.264302 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:47.264264 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:47.322812 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:47.322764 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:49.015942 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:49.015905 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:49.117930 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:49.117892 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:53.656690 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:53.656596 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:53.921403 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:53.921310 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:54.388394 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:54.388356 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f4464dd95-gw85n_9548884b-0ad9-404a-a7bb-cd8dd672957d/authorino/0.log" Jul 14 16:08:58.411839 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:58.411796 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:08:58.463878 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:08:58.463830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:09:00.060301 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:00.060273 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ddsx7_6f993324-9ef2-4b07-8238-4716dc18ea75/server/0.log" Jul 14 16:09:00.182044 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:00.182011 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-6tgk4_dc512517-286f-46e6-b332-34c74d649ed1/manager/0.log" Jul 14 16:09:00.309633 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:00.309551 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66c4b7dd84-62w2b_027d63ac-3923-4ea6-bd02-c2d7f45de26d/copy-manifests/0.log" Jul 14 16:09:00.318896 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:00.318871 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66c4b7dd84-62w2b_027d63ac-3923-4ea6-bd02-c2d7f45de26d/manager/0.log" Jul 14 16:09:01.561415 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:01.561384 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-768d39-gpcbk_163004ce-86d4-4d9f-8fbe-861c1fd4cfda/revoke-keys/0.log" Jul 14 16:09:01.674493 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:01.674461 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-0e29de-nhv9h_6d6115b3-f640-4003-9754-80166826250c/revoke-keys/0.log" Jul 14 16:09:02.023088 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:02.023055 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-81addd-vd6z6_931a3f4e-484e-4751-8237-8544ef36df1e/revoke-keys/0.log" Jul 14 16:09:02.510363 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:02.510331 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-bea800-pwd2v_62a6572d-12ed-41a0-b1dc-be196e8f03dc/revoke-keys/0.log" Jul 14 16:09:02.625010 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:02.624983 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-519447-5sfj4_9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b/revoke-keys/0.log" Jul 14 16:09:03.572098 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:03.572062 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr_da3c5761-0462-4689-b1ea-968bb6c0d5de/util/0.log" Jul 14 16:09:03.580324 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:03.580301 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr_da3c5761-0462-4689-b1ea-968bb6c0d5de/pull/0.log" Jul 14 16:09:03.588689 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:03.588671 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr_da3c5761-0462-4689-b1ea-968bb6c0d5de/extract/0.log" Jul 14 16:09:03.706948 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:03.706919 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv_83061462-6051-466b-9e61-90164f652346/util/0.log" Jul 14 16:09:03.714837 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:03.714813 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv_83061462-6051-466b-9e61-90164f652346/pull/0.log" Jul 14 16:09:03.722974 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:03.722953 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv_83061462-6051-466b-9e61-90164f652346/extract/0.log" Jul 14 16:09:03.836632 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:03.836548 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5_cb58ab6d-8105-41a1-9c2f-6d74e1010daa/util/0.log" Jul 14 16:09:03.844698 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:03.844666 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5_cb58ab6d-8105-41a1-9c2f-6d74e1010daa/pull/0.log" Jul 14 16:09:03.852883 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:03.852864 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5_cb58ab6d-8105-41a1-9c2f-6d74e1010daa/extract/0.log" Jul 14 16:09:03.968962 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:03.968928 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c_bc600c37-b36e-4d51-9160-f91d1e881813/util/0.log" Jul 14 16:09:03.977610 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:03.977581 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c_bc600c37-b36e-4d51-9160-f91d1e881813/pull/0.log" Jul 14 16:09:03.985764 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:03.985740 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c_bc600c37-b36e-4d51-9160-f91d1e881813/extract/0.log" Jul 14 16:09:04.115458 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:04.115372 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f4464dd95-gw85n_9548884b-0ad9-404a-a7bb-cd8dd672957d/authorino/0.log" Jul 14 16:09:04.873449 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:04.873419 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-28mc5_0a248343-3e20-4dbf-bac9-d55c09d61474/limitador/0.log" Jul 14 16:09:05.652056 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:05.652023 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f5f96f494-p85qh_dab2d01e-5dd1-4a4c-8054-fa921782c9ea/kube-auth-proxy/0.log" Jul 14 16:09:06.304036 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:06.304001 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cd7778556-bmkdk_4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c/router/0.log" Jul 14 16:09:11.687682 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:11.687644 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zc89m/must-gather-6gj8l"] Jul 14 16:09:11.688134 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:11.688118 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b" containerName="revoke-keys" Jul 14 16:09:11.688203 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:11.688139 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b" containerName="revoke-keys" Jul 14 16:09:11.688249 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:11.688238 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f3a7e85-2fa4-4b98-bc42-bb54f4c7ce1b" containerName="revoke-keys" Jul 14 16:09:11.691072 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:11.691050 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zc89m/must-gather-6gj8l" Jul 14 16:09:11.694216 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:11.694196 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zc89m\"/\"openshift-service-ca.crt\"" Jul 14 16:09:11.694347 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:11.694266 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zc89m\"/\"default-dockercfg-khfzj\"" Jul 14 16:09:11.695783 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:11.695763 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zc89m\"/\"kube-root-ca.crt\"" Jul 14 16:09:11.700102 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:11.700081 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zc89m/must-gather-6gj8l"] Jul 14 16:09:11.712443 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:11.712409 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:09:11.799705 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:11.799674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7f8119a9-3d5b-489c-b84c-c84b00462c2a-must-gather-output\") pod \"must-gather-6gj8l\" (UID: \"7f8119a9-3d5b-489c-b84c-c84b00462c2a\") " pod="openshift-must-gather-zc89m/must-gather-6gj8l" Jul 14 16:09:11.799861 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:11.799735 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmkdt\" (UniqueName: \"kubernetes.io/projected/7f8119a9-3d5b-489c-b84c-c84b00462c2a-kube-api-access-kmkdt\") pod \"must-gather-6gj8l\" (UID: \"7f8119a9-3d5b-489c-b84c-c84b00462c2a\") " pod="openshift-must-gather-zc89m/must-gather-6gj8l" Jul 14 16:09:11.808983 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:11.808952 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" limit=1024 Jul 14 16:09:11.900906 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:11.900858 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmkdt\" (UniqueName: \"kubernetes.io/projected/7f8119a9-3d5b-489c-b84c-c84b00462c2a-kube-api-access-kmkdt\") pod \"must-gather-6gj8l\" (UID: \"7f8119a9-3d5b-489c-b84c-c84b00462c2a\") " pod="openshift-must-gather-zc89m/must-gather-6gj8l" Jul 14 16:09:11.901092 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:11.900969 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7f8119a9-3d5b-489c-b84c-c84b00462c2a-must-gather-output\") pod \"must-gather-6gj8l\" (UID: \"7f8119a9-3d5b-489c-b84c-c84b00462c2a\") " pod="openshift-must-gather-zc89m/must-gather-6gj8l" Jul 14 16:09:11.901337 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:11.901316 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7f8119a9-3d5b-489c-b84c-c84b00462c2a-must-gather-output\") pod \"must-gather-6gj8l\" (UID: \"7f8119a9-3d5b-489c-b84c-c84b00462c2a\") " pod="openshift-must-gather-zc89m/must-gather-6gj8l" Jul 14 16:09:11.910943 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:11.910920 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmkdt\" (UniqueName: \"kubernetes.io/projected/7f8119a9-3d5b-489c-b84c-c84b00462c2a-kube-api-access-kmkdt\") pod \"must-gather-6gj8l\" (UID: \"7f8119a9-3d5b-489c-b84c-c84b00462c2a\") " pod="openshift-must-gather-zc89m/must-gather-6gj8l" Jul 14 16:09:12.002347 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:12.002255 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zc89m/must-gather-6gj8l" Jul 14 16:09:12.143208 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:12.143164 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zc89m/must-gather-6gj8l"] Jul 14 16:09:12.144501 ip-10-0-128-22 kubenswrapper[2579]: W0714 16:09:12.144468 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f8119a9_3d5b_489c_b84c_c84b00462c2a.slice/crio-ff4dc5922220659c0ac944941c1550e576b9cfa474813323b9b951060ff2bf9a WatchSource:0}: Error finding container ff4dc5922220659c0ac944941c1550e576b9cfa474813323b9b951060ff2bf9a: Status 404 returned error can't find the container with id ff4dc5922220659c0ac944941c1550e576b9cfa474813323b9b951060ff2bf9a Jul 14 16:09:12.556216 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:12.556163 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zc89m/must-gather-6gj8l" event={"ID":"7f8119a9-3d5b-489c-b84c-c84b00462c2a","Type":"ContainerStarted","Data":"ff4dc5922220659c0ac944941c1550e576b9cfa474813323b9b951060ff2bf9a"} Jul 14 16:09:13.565572 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:13.565527 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zc89m/must-gather-6gj8l" event={"ID":"7f8119a9-3d5b-489c-b84c-c84b00462c2a","Type":"ContainerStarted","Data":"3a38c8ae7333ad92511d41778fae5cc5cdd3f274a4d1d5d0a90f6f85c4f83db3"} Jul 14 16:09:13.566078 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:13.565601 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zc89m/must-gather-6gj8l" event={"ID":"7f8119a9-3d5b-489c-b84c-c84b00462c2a","Type":"ContainerStarted","Data":"32fd27b1bac29a8719b2e903d07a16658024f27c59b694543ad4544dad88a30b"} Jul 14 16:09:13.584986 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:13.584913 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zc89m/must-gather-6gj8l" podStartSLOduration=1.7623852009999998 podStartE2EDuration="2.584892111s" totalImagesPullingTime="822.50691ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 16:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 16:09:13.583506856 +0000 UTC m=+2894.435890023" watchObservedRunningTime="2026-07-14 16:09:13.584892111 +0000 UTC m=+2894.437275276" Jul 14 16:09:14.524580 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:14.524548 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-785zz_f42d7bd8-f742-4b0a-83d9-2e0f121bda84/global-pull-secret-syncer/0.log" Jul 14 16:09:14.674438 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:14.674397 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-89g6j_36b98868-c196-465a-8561-d10895ae0f3c/konnectivity-agent/0.log" Jul 14 16:09:14.793536 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:14.793436 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-22.ec2.internal_ebf13fbb58bd809e707ce060a950b3f9/haproxy/0.log" Jul 14 16:09:18.760587 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:18.760527 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr_da3c5761-0462-4689-b1ea-968bb6c0d5de/extract/0.log" Jul 14 16:09:18.788663 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:18.788561 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr_da3c5761-0462-4689-b1ea-968bb6c0d5de/util/0.log" Jul 14 16:09:18.811610 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:18.811587 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f7nhr_da3c5761-0462-4689-b1ea-968bb6c0d5de/pull/0.log" Jul 14 16:09:18.855897 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:18.855872 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv_83061462-6051-466b-9e61-90164f652346/extract/0.log" Jul 14 16:09:18.882874 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:18.882843 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv_83061462-6051-466b-9e61-90164f652346/util/0.log" Jul 14 16:09:18.915139 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:18.915103 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qjfhv_83061462-6051-466b-9e61-90164f652346/pull/0.log" Jul 14 16:09:18.949941 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:18.949915 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5_cb58ab6d-8105-41a1-9c2f-6d74e1010daa/extract/0.log" Jul 14 16:09:18.976384 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:18.976353 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5_cb58ab6d-8105-41a1-9c2f-6d74e1010daa/util/0.log" Jul 14 16:09:19.017294 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:19.017145 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kt4l5_cb58ab6d-8105-41a1-9c2f-6d74e1010daa/pull/0.log" Jul 14 16:09:19.049941 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:19.049908 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c_bc600c37-b36e-4d51-9160-f91d1e881813/extract/0.log" Jul 14 16:09:19.074878 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:19.074813 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c_bc600c37-b36e-4d51-9160-f91d1e881813/util/0.log" Jul 14 16:09:19.100468 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:19.099682 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hzv2c_bc600c37-b36e-4d51-9160-f91d1e881813/pull/0.log" Jul 14 16:09:19.515783 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:19.515735 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f4464dd95-gw85n_9548884b-0ad9-404a-a7bb-cd8dd672957d/authorino/0.log" Jul 14 16:09:19.950321 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:19.950244 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-28mc5_0a248343-3e20-4dbf-bac9-d55c09d61474/limitador/0.log" Jul 14 16:09:22.110449 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:22.110420 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-96wbh_dd4a3f8d-bceb-4bff-ac61-3b3bc77e28c2/cluster-monitoring-operator/0.log" Jul 14 16:09:22.510051 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:22.509912 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8hrpm_1460f341-5423-4590-aad7-7bde6bbefe61/node-exporter/0.log" Jul 14 16:09:22.534276 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:22.534245 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8hrpm_1460f341-5423-4590-aad7-7bde6bbefe61/kube-rbac-proxy/0.log" Jul 14 16:09:22.560159 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:22.560128 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8hrpm_1460f341-5423-4590-aad7-7bde6bbefe61/init-textfile/0.log" Jul 14 16:09:24.272146 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.272115 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-4xn55_02191573-3edc-4fe3-97af-d478beb7575c/networking-console-plugin/0.log" Jul 14 16:09:24.618796 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.618758 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh"] Jul 14 16:09:24.625121 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.625086 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" Jul 14 16:09:24.632413 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.632384 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh"] Jul 14 16:09:24.733725 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.733686 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d-sys\") pod \"perf-node-gather-daemonset-2h2lh\" (UID: \"fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" Jul 14 16:09:24.733921 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.733735 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkzsz\" (UniqueName: \"kubernetes.io/projected/fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d-kube-api-access-vkzsz\") pod \"perf-node-gather-daemonset-2h2lh\" (UID: \"fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" Jul 14 16:09:24.733921 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.733827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d-proc\") pod \"perf-node-gather-daemonset-2h2lh\" (UID: \"fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" Jul 14 16:09:24.733921 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.733858 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d-lib-modules\") pod \"perf-node-gather-daemonset-2h2lh\" (UID: \"fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" Jul 14 16:09:24.734108 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.733941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d-podres\") pod \"perf-node-gather-daemonset-2h2lh\" (UID: \"fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" Jul 14 16:09:24.835250 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.835206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d-proc\") pod \"perf-node-gather-daemonset-2h2lh\" (UID: \"fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" Jul 14 16:09:24.835250 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.835248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d-lib-modules\") pod \"perf-node-gather-daemonset-2h2lh\" (UID: \"fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" Jul 14 16:09:24.835532 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.835294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d-podres\") pod \"perf-node-gather-daemonset-2h2lh\" (UID: \"fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" Jul 14 16:09:24.835532 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.835319 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d-proc\") pod \"perf-node-gather-daemonset-2h2lh\" (UID: \"fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" Jul 14 16:09:24.835532 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.835441 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d-podres\") pod \"perf-node-gather-daemonset-2h2lh\" (UID: \"fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" Jul 14 16:09:24.835701 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.835441 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d-lib-modules\") pod \"perf-node-gather-daemonset-2h2lh\" (UID: \"fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" Jul 14 16:09:24.835869 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.835839 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d-sys\") pod \"perf-node-gather-daemonset-2h2lh\" (UID: \"fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" Jul 14 16:09:24.835966 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.835901 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkzsz\" (UniqueName: \"kubernetes.io/projected/fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d-kube-api-access-vkzsz\") pod \"perf-node-gather-daemonset-2h2lh\" (UID: \"fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" Jul 14 16:09:24.840196 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.836240 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d-sys\") pod \"perf-node-gather-daemonset-2h2lh\" (UID: \"fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" Jul 14 16:09:24.851623 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.851128 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkzsz\" (UniqueName: \"kubernetes.io/projected/fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d-kube-api-access-vkzsz\") pod \"perf-node-gather-daemonset-2h2lh\" (UID: \"fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" Jul 14 16:09:24.947433 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.947356 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" Jul 14 16:09:24.973542 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.973502 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/1.log" Jul 14 16:09:24.979597 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:24.979542 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-b9cq8_0c30c26a-ed99-46f7-89d8-020e9ed3a7c8/console-operator/2.log" Jul 14 16:09:25.151341 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:25.151217 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh"] Jul 14 16:09:25.669234 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:25.669160 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" event={"ID":"fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d","Type":"ContainerStarted","Data":"8ded9090e4ee44aeab76bc1d6b8aad24b07123e72c3109c441318589aac34174"} Jul 14 16:09:25.669739 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:25.669264 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" event={"ID":"fab5a9a3-e7b8-4a38-a06a-ac76d81a0e7d","Type":"ContainerStarted","Data":"27270473f24ef2522c0b831dc0714f3e363ba357ac920354c70be9e9255aaf97"} Jul 14 16:09:25.670125 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:25.670102 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" Jul 14 16:09:25.689874 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:25.689818 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" podStartSLOduration=1.689798749 podStartE2EDuration="1.689798749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 16:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 16:09:25.68804979 +0000 UTC m=+2906.540432956" watchObservedRunningTime="2026-07-14 16:09:25.689798749 +0000 UTC m=+2906.542181914" Jul 14 16:09:26.222720 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:26.222680 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-l52n9_99421093-ab46-4941-9ffe-f198f2d8b178/volume-data-source-validator/0.log" Jul 14 16:09:27.214871 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:27.214845 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lznzq_d94ec422-3618-4b68-ae06-fc9232319c6d/dns/0.log" Jul 14 16:09:27.239865 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:27.239832 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lznzq_d94ec422-3618-4b68-ae06-fc9232319c6d/kube-rbac-proxy/0.log" Jul 14 16:09:27.385189 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:27.385148 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jj7cz_db4c8e2e-aa46-4542-b5d8-8401fa2dfbeb/dns-node-resolver/0.log" Jul 14 16:09:27.939897 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:27.939865 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-574655b545-ggbxf_28a71623-814f-4087-8206-4b0baca8e043/registry/0.log" Jul 14 16:09:28.012669 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:28.012638 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-km9wq_3f3999dd-ca70-4115-8778-3c73e9f14124/node-ca/0.log" Jul 14 16:09:29.193486 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:29.193448 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f5f96f494-p85qh_dab2d01e-5dd1-4a4c-8054-fa921782c9ea/kube-auth-proxy/0.log" Jul 14 16:09:29.416722 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:29.416686 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cd7778556-bmkdk_4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c/router/0.log" Jul 14 16:09:30.159049 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:30.159009 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xrzdx_f022e7cf-42aa-4780-b3f0-6004f70ae107/serve-healthcheck-canary/0.log" Jul 14 16:09:30.651906 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:30.651876 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-zxnts_cbb50f85-a13f-451d-a4bb-df7a9f212438/insights-operator/0.log" Jul 14 16:09:30.653062 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:30.653040 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-zxnts_cbb50f85-a13f-451d-a4bb-df7a9f212438/insights-operator/1.log" Jul 14 16:09:30.822353 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:30.822325 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xmsfd_28db3bec-425f-408b-b575-11bbe25511b7/kube-rbac-proxy/0.log" Jul 14 16:09:30.844652 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:30.844628 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xmsfd_28db3bec-425f-408b-b575-11bbe25511b7/exporter/0.log" Jul 14 16:09:30.867364 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:30.867337 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xmsfd_28db3bec-425f-408b-b575-11bbe25511b7/extractor/0.log" Jul 14 16:09:32.694191 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:32.694151 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-2h2lh" Jul 14 16:09:33.375436 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:33.375408 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ddsx7_6f993324-9ef2-4b07-8238-4716dc18ea75/server/0.log" Jul 14 16:09:33.467611 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:33.467581 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-6tgk4_dc512517-286f-46e6-b332-34c74d649ed1/manager/0.log" Jul 14 16:09:33.494801 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:33.494774 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66c4b7dd84-62w2b_027d63ac-3923-4ea6-bd02-c2d7f45de26d/manager/0.log" Jul 14 16:09:33.517865 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:33.517842 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66c4b7dd84-62w2b_027d63ac-3923-4ea6-bd02-c2d7f45de26d/copy-manifests/0.log" Jul 14 16:09:34.935213 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:34.935186 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f5f96f494-p85qh_dab2d01e-5dd1-4a4c-8054-fa921782c9ea/kube-auth-proxy/0.log" Jul 14 16:09:35.022055 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:35.022027 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-d87d6d9c6-qxw67_215eeffa-ada4-475d-b763-dedbee5cc0fc/manager/0.log" Jul 14 16:09:35.054537 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:35.054497 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-gd2t7_9798b5f6-2aec-4fa1-9ff3-f93ccc1493e5/openshift-lws-operator/0.log" Jul 14 16:09:35.129280 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:35.129250 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cd7778556-bmkdk_4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c/router/0.log" Jul 14 16:09:36.700674 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:36.700638 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f5f96f494-p85qh_dab2d01e-5dd1-4a4c-8054-fa921782c9ea/kube-auth-proxy/0.log" Jul 14 16:09:36.889647 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:36.889607 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cd7778556-bmkdk_4a90d0a9-4d08-4c0d-abfe-dadd776ecc8c/router/0.log" Jul 14 16:09:40.146625 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:40.146585 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-jhjxc_cb4ec4df-6611-41c7-b1aa-f32a58b3a251/migrator/0.log" Jul 14 16:09:40.172940 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:40.172912 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-jhjxc_cb4ec4df-6611-41c7-b1aa-f32a58b3a251/graceful-termination/0.log" Jul 14 16:09:40.531606 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:40.531512 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-jkhlz_2d1490f8-b484-46ee-83d6-4bb7fae67fd8/kube-storage-version-migrator-operator/1.log" Jul 14 16:09:40.532470 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:40.532453 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-jkhlz_2d1490f8-b484-46ee-83d6-4bb7fae67fd8/kube-storage-version-migrator-operator/0.log" Jul 14 16:09:41.475603 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:41.475569 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-84dvv_d8339b58-8a56-4e52-8722-52249ad123d3/kube-multus/0.log" Jul 14 16:09:41.927232 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:41.927197 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvkjc_5062eb6e-d839-4164-aad5-057e03ade017/kube-multus-additional-cni-plugins/0.log" Jul 14 16:09:41.950986 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:41.950957 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvkjc_5062eb6e-d839-4164-aad5-057e03ade017/egress-router-binary-copy/0.log" Jul 14 16:09:41.976034 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:41.976005 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvkjc_5062eb6e-d839-4164-aad5-057e03ade017/cni-plugins/0.log" Jul 14 16:09:42.001547 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:42.001530 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvkjc_5062eb6e-d839-4164-aad5-057e03ade017/bond-cni-plugin/0.log" Jul 14 16:09:42.024723 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:42.024702 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvkjc_5062eb6e-d839-4164-aad5-057e03ade017/routeoverride-cni/0.log" Jul 14 16:09:42.048362 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:42.048343 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvkjc_5062eb6e-d839-4164-aad5-057e03ade017/whereabouts-cni-bincopy/0.log" Jul 14 16:09:42.070846 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:42.070826 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvkjc_5062eb6e-d839-4164-aad5-057e03ade017/whereabouts-cni/0.log" Jul 14 16:09:42.280149 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:42.280063 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kg2pm_edc388f1-89b5-45bf-9672-3e1a135fed4e/network-metrics-daemon/0.log" Jul 14 16:09:42.301905 ip-10-0-128-22 kubenswrapper[2579]: I0714 16:09:42.301877 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kg2pm_edc388f1-89b5-45bf-9672-3e1a135fed4e/kube-rbac-proxy/0.log"