Jul 07 18:22:56.488680 ip-10-0-141-108 systemd[1]: Starting Kubernetes Kubelet... Jul 07 18:22:56.929738 ip-10-0-141-108 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 18:22:56.929738 ip-10-0-141-108 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 18:22:56.929738 ip-10-0-141-108 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 18:22:56.929738 ip-10-0-141-108 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 18:22:56.929738 ip-10-0-141-108 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 18:22:56.931897 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.931693 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 18:22:56.937672 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937647 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 18:22:56.937672 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937669 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 18:22:56.937672 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937673 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 18:22:56.937672 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937676 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 07 18:22:56.937838 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937682 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 18:22:56.937838 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937685 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 18:22:56.937838 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937688 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 18:22:56.937838 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937692 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 18:22:56.937838 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937695 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 18:22:56.937838 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937698 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 18:22:56.937838 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937700 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 18:22:56.937838 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937703 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 18:22:56.937838 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937706 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 18:22:56.937838 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937709 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 18:22:56.937838 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937711 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 18:22:56.937838 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937715 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 18:22:56.937838 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937720 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 18:22:56.937838 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937730 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 18:22:56.937838 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937734 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 18:22:56.937838 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937737 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 18:22:56.937838 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937741 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 18:22:56.937838 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937744 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 18:22:56.937838 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937747 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 18:22:56.938311 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937750 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 18:22:56.938311 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937754 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 18:22:56.938311 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937757 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 18:22:56.938311 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937760 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 18:22:56.938311 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937762 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 18:22:56.938311 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937765 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 18:22:56.938311 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937767 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 18:22:56.938311 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937770 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 18:22:56.938311 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937773 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 18:22:56.938311 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937775 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 18:22:56.938311 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937778 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 18:22:56.938311 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937781 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 18:22:56.938311 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937785 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 18:22:56.938311 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937787 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 18:22:56.938311 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937790 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 18:22:56.938311 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937796 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 18:22:56.938311 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937800 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 18:22:56.938311 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937803 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 18:22:56.938311 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937806 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 18:22:56.938311 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937809 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 18:22:56.938802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937811 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 18:22:56.938802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937814 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 18:22:56.938802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937817 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 18:22:56.938802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937819 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 18:22:56.938802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937822 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 18:22:56.938802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937825 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 18:22:56.938802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937828 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 18:22:56.938802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937831 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 18:22:56.938802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937833 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 18:22:56.938802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937836 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 18:22:56.938802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937839 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 18:22:56.938802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937842 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 18:22:56.938802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937845 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 18:22:56.938802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937848 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 18:22:56.938802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937851 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 18:22:56.938802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937854 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 18:22:56.938802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937857 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 18:22:56.938802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937859 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 18:22:56.938802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937862 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 18:22:56.938802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937865 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 18:22:56.939266 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937867 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 18:22:56.939266 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937870 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 18:22:56.939266 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937873 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 18:22:56.939266 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937875 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 18:22:56.939266 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937878 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 18:22:56.939266 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937880 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 18:22:56.939266 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937883 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 18:22:56.939266 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937886 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 18:22:56.939266 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937889 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 18:22:56.939266 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937893 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 18:22:56.939266 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937897 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 18:22:56.939266 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937900 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 18:22:56.939266 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937903 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 18:22:56.939266 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937905 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 18:22:56.939266 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937908 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 18:22:56.939266 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937910 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 18:22:56.939266 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937912 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 18:22:56.939266 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937915 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 18:22:56.939266 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937917 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 18:22:56.939266 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937928 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 18:22:56.939748 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937931 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 18:22:56.939748 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937933 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 18:22:56.939748 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937936 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 18:22:56.939748 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937940 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 18:22:56.939748 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937943 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 18:22:56.939748 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937946 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 18:22:56.939748 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937948 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 18:22:56.939748 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937951 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 18:22:56.939748 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937954 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 18:22:56.939748 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937957 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 18:22:56.939748 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937959 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 18:22:56.939748 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937962 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 18:22:56.939748 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937965 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 18:22:56.939748 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937968 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 18:22:56.939748 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937971 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 18:22:56.939748 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.937973 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 18:22:56.939748 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938802 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 18:22:56.939748 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938810 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 18:22:56.939748 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938814 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 18:22:56.939748 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938817 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 18:22:56.940263 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938820 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 18:22:56.940263 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938823 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 18:22:56.940263 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938826 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 18:22:56.940263 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938830 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 18:22:56.940263 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938832 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 18:22:56.940263 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938835 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 18:22:56.940263 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938838 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 18:22:56.940263 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938841 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 18:22:56.940263 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938843 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 18:22:56.940263 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938846 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 18:22:56.940263 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938849 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 18:22:56.940263 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938851 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 18:22:56.940263 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938854 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 18:22:56.940263 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938857 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 18:22:56.940263 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938860 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 18:22:56.940263 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938863 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 18:22:56.940263 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938866 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 18:22:56.940263 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938868 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 18:22:56.940263 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938872 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 18:22:56.940263 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938875 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 18:22:56.940769 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938878 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 18:22:56.940769 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938880 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 18:22:56.940769 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938883 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 18:22:56.940769 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938886 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 18:22:56.940769 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938888 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 18:22:56.940769 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938891 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 18:22:56.940769 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938894 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 18:22:56.940769 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938896 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 07 18:22:56.940769 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938899 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 18:22:56.940769 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938902 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 18:22:56.940769 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938904 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 18:22:56.940769 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938907 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 18:22:56.940769 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938909 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 18:22:56.940769 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938912 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 18:22:56.940769 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938915 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 18:22:56.940769 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938919 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 18:22:56.940769 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938923 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 18:22:56.940769 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938926 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 18:22:56.940769 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938930 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 18:22:56.941227 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938933 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 18:22:56.941227 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938935 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 18:22:56.941227 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938938 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 18:22:56.941227 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938941 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 18:22:56.941227 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938943 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 18:22:56.941227 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938946 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 18:22:56.941227 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938948 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 18:22:56.941227 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938950 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 18:22:56.941227 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938953 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 18:22:56.941227 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938955 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 18:22:56.941227 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938958 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 18:22:56.941227 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938961 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 18:22:56.941227 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938964 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 18:22:56.941227 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938966 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 18:22:56.941227 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938969 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 18:22:56.941227 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938972 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 18:22:56.941227 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938975 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 18:22:56.941227 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938978 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 18:22:56.941227 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938980 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 18:22:56.941227 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938983 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 18:22:56.941725 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938985 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 18:22:56.941725 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938988 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 18:22:56.941725 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938990 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 18:22:56.941725 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938993 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 18:22:56.941725 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.938997 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 18:22:56.941725 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939001 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 18:22:56.941725 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939004 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 18:22:56.941725 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939007 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 18:22:56.941725 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939010 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 18:22:56.941725 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939014 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 18:22:56.941725 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939017 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 18:22:56.941725 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939021 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 18:22:56.941725 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939024 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 18:22:56.941725 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939026 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 18:22:56.941725 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939030 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 18:22:56.941725 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939032 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 18:22:56.941725 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939035 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 18:22:56.941725 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939038 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 18:22:56.941725 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939040 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 18:22:56.942202 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939043 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 18:22:56.942202 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939045 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 18:22:56.942202 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939048 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 18:22:56.942202 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939051 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 18:22:56.942202 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939054 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 18:22:56.942202 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939057 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 18:22:56.942202 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939060 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 18:22:56.942202 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939063 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 18:22:56.942202 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939065 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 18:22:56.942202 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939068 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 18:22:56.942202 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939070 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 18:22:56.942202 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939073 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 18:22:56.942202 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939075 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 18:22:56.942202 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939078 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 18:22:56.942202 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939080 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 18:22:56.942202 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939083 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 18:22:56.942202 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939086 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 18:22:56.942202 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939547 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 18:22:56.942202 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939553 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 18:22:56.942202 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939556 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 18:22:56.942696 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939560 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 18:22:56.942696 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939564 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 18:22:56.942696 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939566 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 18:22:56.942696 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939569 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 18:22:56.942696 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939572 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 18:22:56.942696 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939575 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 18:22:56.942696 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939578 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 18:22:56.942696 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939581 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 18:22:56.942696 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939583 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 18:22:56.942696 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939586 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 18:22:56.942696 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939588 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 18:22:56.942696 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939591 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 18:22:56.942696 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939594 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 18:22:56.942696 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939597 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 18:22:56.942696 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939601 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 18:22:56.942696 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939604 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 18:22:56.942696 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939607 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 18:22:56.942696 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939610 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 18:22:56.942696 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939613 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 18:22:56.943145 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939615 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 18:22:56.943145 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939618 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 18:22:56.943145 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939621 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 18:22:56.943145 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939623 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 18:22:56.943145 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939626 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 18:22:56.943145 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939628 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 18:22:56.943145 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939631 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 18:22:56.943145 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939634 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 18:22:56.943145 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939636 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 18:22:56.943145 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939640 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 18:22:56.943145 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939642 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 18:22:56.943145 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939645 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 18:22:56.943145 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939647 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 18:22:56.943145 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939650 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 18:22:56.943145 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939652 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 18:22:56.943145 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939656 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 18:22:56.943145 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939658 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 18:22:56.943145 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939660 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 18:22:56.943145 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939663 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 18:22:56.943145 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939666 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 18:22:56.943681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939670 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 18:22:56.943681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939673 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 18:22:56.943681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939675 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 18:22:56.943681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939678 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 18:22:56.943681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939681 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 18:22:56.943681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939684 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 18:22:56.943681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939686 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 18:22:56.943681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939689 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 18:22:56.943681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939691 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 07 18:22:56.943681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939694 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 18:22:56.943681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939696 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 18:22:56.943681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939699 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 18:22:56.943681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939701 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 18:22:56.943681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939704 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 18:22:56.943681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939707 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 18:22:56.943681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939709 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 18:22:56.943681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939711 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 18:22:56.943681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939714 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 18:22:56.943681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939716 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 18:22:56.943681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939719 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 18:22:56.944176 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939721 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 18:22:56.944176 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939727 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 18:22:56.944176 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939730 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 18:22:56.944176 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939733 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 18:22:56.944176 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939735 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 18:22:56.944176 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939738 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 18:22:56.944176 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939741 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 18:22:56.944176 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939743 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 18:22:56.944176 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939746 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 18:22:56.944176 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939749 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 18:22:56.944176 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939751 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 18:22:56.944176 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939754 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 18:22:56.944176 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939757 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 18:22:56.944176 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939760 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 18:22:56.944176 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939763 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 18:22:56.944176 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939768 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 18:22:56.944176 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939771 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 18:22:56.944176 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939774 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 18:22:56.944176 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939777 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939780 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939783 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939785 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939788 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939791 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939793 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939796 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939799 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939801 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939803 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939806 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939809 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939811 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939814 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939818 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939820 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939823 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.939826 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939898 2578 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939905 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 18:22:56.944646 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939912 2578 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939916 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939921 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939924 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939931 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939937 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939942 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939947 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939952 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939957 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939961 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939966 2578 flags.go:64] FLAG: --cgroup-root="" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939971 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939975 2578 flags.go:64] FLAG: --client-ca-file="" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939977 2578 flags.go:64] FLAG: --cloud-provider="external" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939981 2578 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939987 2578 flags.go:64] FLAG: --cluster-domain="" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939990 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939993 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.939996 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940000 2578 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940004 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940007 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 18:22:56.945146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940011 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940014 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940017 2578 flags.go:64] FLAG: --contention-profiling="false" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940021 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940024 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940028 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940031 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940035 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940038 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940041 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940044 2578 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940047 2578 flags.go:64] FLAG: --enable-server="true" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940054 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940059 2578 flags.go:64] FLAG: --event-burst="100" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940062 2578 flags.go:64] FLAG: --event-qps="50" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940065 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940068 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940071 2578 flags.go:64] FLAG: --eviction-hard="" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940075 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940077 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940080 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940085 2578 flags.go:64] FLAG: --eviction-soft="" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940088 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940092 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940095 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 18:22:56.945730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940098 2578 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940101 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940104 2578 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940107 2578 flags.go:64] FLAG: --feature-gates="" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940113 2578 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940116 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940120 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940123 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940126 2578 flags.go:64] FLAG: --healthz-port="10248" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940129 2578 flags.go:64] FLAG: --help="false" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940137 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-141-108.ec2.internal" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940140 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940143 2578 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940147 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940150 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940154 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940157 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940160 2578 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940162 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940165 2578 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940169 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940172 2578 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940175 2578 flags.go:64] FLAG: --kube-reserved="" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940178 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 18:22:56.946359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940181 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940184 2578 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940187 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940189 2578 flags.go:64] FLAG: --lock-file="" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940194 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940197 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940200 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940206 2578 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940209 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940212 2578 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940215 2578 flags.go:64] FLAG: --logging-format="text" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940218 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940221 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940224 2578 flags.go:64] FLAG: --manifest-url="" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940227 2578 flags.go:64] FLAG: --manifest-url-header="" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940231 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940235 2578 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940239 2578 flags.go:64] FLAG: --max-pods="110" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940244 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940247 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940250 2578 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940253 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940256 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940260 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940263 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 18:22:56.946956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940271 2578 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940274 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940277 2578 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940281 2578 flags.go:64] FLAG: --pod-cidr="" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940284 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940290 2578 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940293 2578 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940296 2578 flags.go:64] FLAG: --pods-per-core="0" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940299 2578 flags.go:64] FLAG: --port="10250" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940302 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940305 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09e488ed29a7b4949" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940309 2578 flags.go:64] FLAG: --qos-reserved="" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940313 2578 flags.go:64] FLAG: --read-only-port="10255" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940328 2578 flags.go:64] FLAG: --register-node="true" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940331 2578 flags.go:64] FLAG: --register-with-taints="" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940335 2578 flags.go:64] FLAG: --registry-burst="10" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940338 2578 flags.go:64] FLAG: --registry-qps="5" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940341 2578 flags.go:64] FLAG: --reserved-cpus="" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940345 2578 flags.go:64] FLAG: --reserved-memory="" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940349 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940352 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940355 2578 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940358 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940361 2578 flags.go:64] FLAG: --runonce="false" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940364 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 18:22:56.947571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940368 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940371 2578 flags.go:64] FLAG: --seccomp-default="false" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940374 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940377 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940380 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940383 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940386 2578 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940389 2578 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940392 2578 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940395 2578 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940398 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940401 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940405 2578 flags.go:64] FLAG: --system-cgroups="" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940408 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940414 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940417 2578 flags.go:64] FLAG: --tls-cert-file="" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940420 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940424 2578 flags.go:64] FLAG: --tls-min-version="" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940427 2578 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940432 2578 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940435 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940438 2578 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940441 2578 flags.go:64] FLAG: --v="2" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940446 2578 flags.go:64] FLAG: --version="false" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940450 2578 flags.go:64] FLAG: --vmodule="" Jul 07 18:22:56.948182 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940454 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 18:22:56.948778 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940457 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 18:22:56.948778 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940576 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 18:22:56.948778 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940580 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 18:22:56.948778 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940583 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 18:22:56.948778 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940586 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 18:22:56.948778 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940589 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 18:22:56.948778 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940592 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 18:22:56.948778 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940594 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 18:22:56.948778 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940597 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 18:22:56.948778 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940600 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 18:22:56.948778 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940602 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 18:22:56.948778 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940605 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 18:22:56.948778 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940607 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 18:22:56.948778 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940610 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 18:22:56.948778 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940612 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 18:22:56.948778 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940616 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 18:22:56.948778 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940618 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 18:22:56.948778 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940621 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 18:22:56.948778 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940623 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 18:22:56.948778 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940626 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 18:22:56.949255 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940629 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 18:22:56.949255 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940631 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 18:22:56.949255 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940633 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 18:22:56.949255 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940636 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 18:22:56.949255 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940639 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 18:22:56.949255 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940643 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 18:22:56.949255 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940646 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 18:22:56.949255 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940648 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 18:22:56.949255 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940651 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 18:22:56.949255 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940653 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 18:22:56.949255 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940656 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 18:22:56.949255 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940659 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 18:22:56.949255 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940662 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 18:22:56.949255 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940665 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 18:22:56.949255 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940667 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 18:22:56.949255 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940670 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 18:22:56.949255 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940672 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 18:22:56.949255 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940675 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 18:22:56.949255 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940678 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 18:22:56.949255 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940680 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 18:22:56.949802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940683 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 18:22:56.949802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940687 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 18:22:56.949802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940689 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 18:22:56.949802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940692 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 18:22:56.949802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940695 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 18:22:56.949802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940697 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 18:22:56.949802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940700 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 18:22:56.949802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940702 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 18:22:56.949802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940705 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 18:22:56.949802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940707 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 18:22:56.949802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940710 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 18:22:56.949802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940712 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 18:22:56.949802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940715 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 18:22:56.949802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940718 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 18:22:56.949802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940720 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 18:22:56.949802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940723 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 18:22:56.949802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940726 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 18:22:56.949802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940730 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 18:22:56.949802 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940734 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 18:22:56.950310 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940738 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 18:22:56.950310 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940741 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 18:22:56.950310 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940744 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 18:22:56.950310 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940747 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 18:22:56.950310 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940749 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 18:22:56.950310 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940752 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 18:22:56.950310 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940755 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 18:22:56.950310 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940757 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 18:22:56.950310 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940760 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 18:22:56.950310 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940762 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 18:22:56.950310 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940765 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 18:22:56.950310 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940768 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 18:22:56.950310 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940770 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 18:22:56.950310 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940773 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 18:22:56.950310 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940776 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 18:22:56.950310 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940779 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 18:22:56.950310 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940782 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 18:22:56.950310 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940784 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 18:22:56.950310 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940787 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 07 18:22:56.950310 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940789 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 18:22:56.950811 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940792 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 18:22:56.950811 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940794 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 18:22:56.950811 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940797 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 18:22:56.950811 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940800 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 18:22:56.950811 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940804 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 18:22:56.950811 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940807 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 18:22:56.950811 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940810 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 18:22:56.950811 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940813 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 18:22:56.950811 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940817 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 18:22:56.950811 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940821 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 18:22:56.950811 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940826 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 18:22:56.950811 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940828 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 18:22:56.950811 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940831 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 18:22:56.950811 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940833 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 18:22:56.950811 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940836 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 18:22:56.950811 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940840 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 18:22:56.950811 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940844 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 18:22:56.950811 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940847 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 18:22:56.950811 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940849 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 18:22:56.951276 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940852 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 18:22:56.951276 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.940854 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 18:22:56.951276 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.940860 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 18:22:56.951276 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.948351 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 18:22:56.951276 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.948375 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 18:22:56.951276 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.948398 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 18:22:56.951276 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.948404 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 18:22:56.951276 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948487 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 18:22:56.951276 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948492 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 18:22:56.951276 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948495 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 18:22:56.951276 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948498 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 18:22:56.951276 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948501 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 18:22:56.951276 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948503 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 18:22:56.951276 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948507 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 18:22:56.951276 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948510 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 18:22:56.951700 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948513 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 18:22:56.951700 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948517 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 18:22:56.951700 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948522 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 18:22:56.951700 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948525 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 18:22:56.951700 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948529 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 07 18:22:56.951700 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948531 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 18:22:56.951700 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948534 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 18:22:56.951700 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948537 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 18:22:56.951700 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948539 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 18:22:56.951700 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948542 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 18:22:56.951700 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948546 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 18:22:56.951700 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948548 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 18:22:56.951700 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948551 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 18:22:56.951700 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948553 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 18:22:56.951700 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948555 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 18:22:56.951700 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948558 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 18:22:56.951700 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948560 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 18:22:56.951700 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948563 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 18:22:56.951700 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948566 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 18:22:56.952461 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948569 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 18:22:56.952461 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948572 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 18:22:56.952461 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948582 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 18:22:56.952461 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948585 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 18:22:56.952461 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948589 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 18:22:56.952461 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948592 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 18:22:56.952461 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948595 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 18:22:56.952461 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948598 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 18:22:56.952461 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948601 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 18:22:56.952461 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948604 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 18:22:56.952461 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948606 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 18:22:56.952461 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948609 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 18:22:56.952461 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948612 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 18:22:56.952461 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948615 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 18:22:56.952461 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948617 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 18:22:56.952461 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948620 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 18:22:56.952461 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948622 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 18:22:56.952461 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948625 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 18:22:56.952461 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948627 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 18:22:56.953060 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948630 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 18:22:56.953060 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948632 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 18:22:56.953060 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948635 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 18:22:56.953060 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948638 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 18:22:56.953060 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948640 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 18:22:56.953060 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948643 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 18:22:56.953060 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948645 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 18:22:56.953060 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948647 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 18:22:56.953060 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948650 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 18:22:56.953060 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948653 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 18:22:56.953060 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948655 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 18:22:56.953060 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948658 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 18:22:56.953060 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948660 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 18:22:56.953060 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948663 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 18:22:56.953060 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948665 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 18:22:56.953060 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948669 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 18:22:56.953060 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948678 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 18:22:56.953060 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948681 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 18:22:56.953060 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948683 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 18:22:56.953060 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948686 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 18:22:56.953578 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948688 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 18:22:56.953578 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948691 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 18:22:56.953578 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948693 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 18:22:56.953578 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948696 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 18:22:56.953578 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948699 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 18:22:56.953578 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948701 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 18:22:56.953578 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948704 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 18:22:56.953578 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948706 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 18:22:56.953578 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948709 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 18:22:56.953578 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948711 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 18:22:56.953578 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948714 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 18:22:56.953578 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948716 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 18:22:56.953578 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948719 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 18:22:56.953578 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948721 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 18:22:56.953578 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948724 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 18:22:56.953578 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948727 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 18:22:56.953578 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948729 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 18:22:56.953578 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948732 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 18:22:56.953578 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948734 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 18:22:56.953578 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948736 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 18:22:56.954052 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948739 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 18:22:56.954052 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948741 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 18:22:56.954052 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948743 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 18:22:56.954052 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948746 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 18:22:56.954052 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948748 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 18:22:56.954052 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948751 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 18:22:56.954052 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948753 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 18:22:56.954052 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948757 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 18:22:56.954052 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948760 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 18:22:56.954052 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948768 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 18:22:56.954052 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948771 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 18:22:56.954052 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948773 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 18:22:56.954052 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948776 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 18:22:56.954052 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.948781 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 18:22:56.954052 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948935 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 18:22:56.954467 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948940 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 18:22:56.954467 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948943 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 18:22:56.954467 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948946 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 18:22:56.954467 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948949 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 18:22:56.954467 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948952 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 18:22:56.954467 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948954 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 18:22:56.954467 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948957 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 18:22:56.954467 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948959 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 18:22:56.954467 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948962 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 18:22:56.954467 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948964 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 18:22:56.954467 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948967 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 18:22:56.954467 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948969 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 18:22:56.954467 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948972 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 18:22:56.954467 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948975 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 18:22:56.954467 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948977 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 18:22:56.954467 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948980 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 18:22:56.954467 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948982 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 18:22:56.954467 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948985 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 18:22:56.954467 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948987 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 18:22:56.954467 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948989 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 18:22:56.954954 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948992 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 18:22:56.954954 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948994 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 18:22:56.954954 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948997 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 18:22:56.954954 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.948999 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 18:22:56.954954 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949003 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 18:22:56.954954 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949006 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 18:22:56.954954 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949009 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 18:22:56.954954 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949018 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 18:22:56.954954 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949021 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 18:22:56.954954 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949023 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 18:22:56.954954 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949026 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 18:22:56.954954 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949028 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 18:22:56.954954 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949031 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 18:22:56.954954 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949033 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 18:22:56.954954 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949036 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 18:22:56.954954 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949038 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 18:22:56.954954 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949040 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 18:22:56.954954 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949043 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 18:22:56.954954 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949045 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 18:22:56.954954 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949048 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 18:22:56.955447 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949050 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 18:22:56.955447 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949052 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 18:22:56.955447 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949055 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 18:22:56.955447 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949057 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 18:22:56.955447 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949060 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 18:22:56.955447 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949062 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 18:22:56.955447 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949064 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 18:22:56.955447 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949068 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 18:22:56.955447 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949072 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 07 18:22:56.955447 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949075 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 18:22:56.955447 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949078 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 18:22:56.955447 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949081 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 18:22:56.955447 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949084 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 18:22:56.955447 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949086 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 18:22:56.955447 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949089 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 18:22:56.955447 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949091 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 18:22:56.955447 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949095 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 18:22:56.955447 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949097 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 18:22:56.955447 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949100 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 18:22:56.955447 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949102 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 18:22:56.955929 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949111 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 18:22:56.955929 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949114 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 18:22:56.955929 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949116 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 18:22:56.955929 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949119 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 18:22:56.955929 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949121 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 18:22:56.955929 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949124 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 18:22:56.955929 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949126 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 18:22:56.955929 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949129 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 18:22:56.955929 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949131 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 18:22:56.955929 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949134 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 18:22:56.955929 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949136 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 18:22:56.955929 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949139 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 18:22:56.955929 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949141 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 18:22:56.955929 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949144 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 18:22:56.955929 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949147 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 18:22:56.955929 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949150 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 18:22:56.955929 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949153 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 18:22:56.955929 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949155 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 18:22:56.955929 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949158 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 18:22:56.956446 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949160 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 18:22:56.956446 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949163 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 18:22:56.956446 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949167 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 18:22:56.956446 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949171 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 18:22:56.956446 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949174 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 18:22:56.956446 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949177 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 18:22:56.956446 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949180 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 18:22:56.956446 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949183 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 18:22:56.956446 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949186 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 18:22:56.956446 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949188 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 18:22:56.956446 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949191 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 18:22:56.956446 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949193 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 18:22:56.956446 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949196 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 18:22:56.956446 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949199 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 18:22:56.956446 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949210 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 18:22:56.956446 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949212 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 18:22:56.956446 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949215 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 18:22:56.956446 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949218 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 18:22:56.956446 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:56.949220 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 18:22:56.956897 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.949225 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 18:22:56.956897 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.949417 2578 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 18:22:56.956897 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.954153 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 18:22:56.956897 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.955827 2578 server.go:1019] "Starting client certificate rotation" Jul 07 18:22:56.956897 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.955930 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 18:22:56.956897 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.955968 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 18:22:56.958512 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.958496 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 18:22:56.962561 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.962536 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 18:22:56.975422 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.975398 2578 log.go:25] "Validated CRI v1 runtime API" Jul 07 18:22:56.977796 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.977779 2578 log.go:25] "Validated CRI v1 image API" Jul 07 18:22:56.979188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.979164 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 18:22:56.981969 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.981939 2578 fs.go:135] Filesystem UUIDs: map[1736229c-ed5b-4ede-9d03-74af5d8e3023:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f53ba87b-36f3-47bc-a1ab-fd8c9df7177b:/dev/nvme0n1p4] Jul 07 18:22:56.982101 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.981965 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 18:22:56.988498 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.988346 2578 manager.go:216] Machine: {Timestamp:2026-07-07 18:22:56.98661486 +0000 UTC m=+0.370408246 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099549 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28c0e86af63c195e851781f834a713 SystemUUID:ec28c0e8-6af6-3c19-5e85-1781f834a713 BootID:ffaf3f7a-b9a2-4601-9b61-6efbed7fe9ec Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:16:c8:d6:7f:d3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:16:c8:d6:7f:d3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:48:fc:86:82:66 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 18:22:56.988498 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.988472 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 18:22:56.988698 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.988611 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 18:22:56.989117 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.989095 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 18:22:56.991908 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.991870 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 18:22:56.992096 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.991914 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-108.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 18:22:56.992814 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.992793 2578 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 18:22:56.992885 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.992820 2578 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 18:22:56.992885 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.992832 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 18:22:56.993438 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.993426 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 18:22:56.994950 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.994935 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 18:22:56.997892 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.997879 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 07 18:22:56.998045 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.998034 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 18:22:56.998076 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.998067 2578 kubelet.go:486] "Attempting to sync node with API server" Jul 07 18:22:56.998103 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.998076 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 18:22:56.998103 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.998096 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 18:22:56.998164 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.998107 2578 kubelet.go:398] "Adding apiserver pod source" Jul 07 18:22:56.998164 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.998117 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 18:22:56.998606 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.998593 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 18:22:56.998650 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:56.998630 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 18:22:57.002634 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.002612 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 18:22:57.004842 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.004828 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 18:22:57.004905 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.004858 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 18:22:57.006084 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.006073 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 18:22:57.006123 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.006091 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 18:22:57.006123 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.006097 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 18:22:57.006123 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.006102 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 18:22:57.006123 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.006112 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 18:22:57.006123 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.006118 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 18:22:57.006123 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.006125 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 18:22:57.006284 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.006131 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 18:22:57.006284 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.006138 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 18:22:57.006284 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.006144 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 18:22:57.006284 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.006153 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 18:22:57.006284 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.006159 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 18:22:57.007152 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.007140 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 18:22:57.007152 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.007153 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 18:22:57.008036 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.008010 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-108.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 18:22:57.008036 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.008010 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 18:22:57.010719 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.010706 2578 server.go:1268] "Started kubelet" Jul 07 18:22:57.010819 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.010790 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 18:22:57.011005 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.010918 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 18:22:57.011095 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.011017 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 18:22:57.011646 ip-10-0-141-108 systemd[1]: Started Kubernetes Kubelet. Jul 07 18:22:57.012435 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.012401 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 18:22:57.013502 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.013475 2578 server.go:310] "Adding debug handlers to kubelet server" Jul 07 18:22:57.016248 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.016225 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vlrcq" Jul 07 18:22:57.016821 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.016788 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 18:22:57.018195 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.018179 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 18:22:57.018247 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.018213 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 18:22:57.018762 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.018419 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-108.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 18:22:57.018762 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.018407 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-108.ec2.internal\" not found" Jul 07 18:22:57.018963 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.018949 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 18:22:57.019028 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.019021 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 18:22:57.019077 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.019035 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 18:22:57.019214 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.019191 2578 reconstruct.go:97] "Volume reconstruction finished" Jul 07 18:22:57.019214 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.019206 2578 reconciler.go:29] "Reconciler: start to sync state" Jul 07 18:22:57.019475 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.018399 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-108.ec2.internal.18c015081be5a9a2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-108.ec2.internal,UID:ip-10-0-141-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-108.ec2.internal,},FirstTimestamp:2026-07-07 18:22:57.010674082 +0000 UTC m=+0.394467468,LastTimestamp:2026-07-07 18:22:57.010674082 +0000 UTC m=+0.394467468,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-108.ec2.internal,}" Jul 07 18:22:57.019601 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.019510 2578 factory.go:153] Registering CRI-O factory Jul 07 18:22:57.019601 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.019523 2578 factory.go:223] Registration of the crio container factory successfully Jul 07 18:22:57.019601 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.019570 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 18:22:57.019601 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.019581 2578 factory.go:55] Registering systemd factory Jul 07 18:22:57.019601 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.019589 2578 factory.go:223] Registration of the systemd container factory successfully Jul 07 18:22:57.019774 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.019608 2578 factory.go:103] Registering Raw factory Jul 07 18:22:57.019774 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.019622 2578 manager.go:1196] Started watching for new ooms in manager Jul 07 18:22:57.020346 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.020315 2578 manager.go:349] Starting recovery of all containers Jul 07 18:22:57.024987 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.024966 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vlrcq" Jul 07 18:22:57.028504 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.028470 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 18:22:57.031442 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.031419 2578 manager.go:354] Recovery completed Jul 07 18:22:57.032128 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.032097 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-108.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 18:22:57.034089 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.033913 2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 07 18:22:57.037240 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.037227 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 18:22:57.039095 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.039080 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasSufficientMemory" Jul 07 18:22:57.039162 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.039113 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 18:22:57.039162 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.039126 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasSufficientPID" Jul 07 18:22:57.039855 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.039835 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 18:22:57.039855 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.039848 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 18:22:57.039961 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.039852 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 18:22:57.039961 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.039923 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 07 18:22:57.040545 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.040527 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasSufficientMemory" Jul 07 18:22:57.040593 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.040562 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 18:22:57.040593 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.040577 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasSufficientPID" Jul 07 18:22:57.041083 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.041073 2578 policy_none.go:49] "None policy: Start" Jul 07 18:22:57.041113 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.041091 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 18:22:57.041113 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.041104 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 18:22:57.042138 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.042126 2578 policy_none.go:47] "Start" Jul 07 18:22:57.089988 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.089969 2578 manager.go:337] "Starting Device Plugin manager" Jul 07 18:22:57.090116 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.090024 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 18:22:57.090116 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.090038 2578 server.go:87] "Starting device plugin registration server" Jul 07 18:22:57.090397 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.090383 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 18:22:57.090482 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.090400 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 18:22:57.090522 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.090496 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 18:22:57.090604 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.090590 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 18:22:57.090646 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.090608 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 18:22:57.091356 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.091299 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 18:22:57.091463 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.091381 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-108.ec2.internal\" not found" Jul 07 18:22:57.189795 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.189703 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 18:22:57.191070 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.190954 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 18:22:57.191070 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.190958 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 18:22:57.191070 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.191005 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 18:22:57.191070 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.191045 2578 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 18:22:57.191269 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.191087 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 18:22:57.191928 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.191903 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasSufficientMemory" Jul 07 18:22:57.192047 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.191939 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 18:22:57.192047 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.191951 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasSufficientPID" Jul 07 18:22:57.192047 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.191978 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-108.ec2.internal" Jul 07 18:22:57.193637 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.193617 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 18:22:57.200725 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.200696 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-108.ec2.internal" Jul 07 18:22:57.200725 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.200724 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-108.ec2.internal\": node \"ip-10-0-141-108.ec2.internal\" not found" Jul 07 18:22:57.246146 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.246107 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-108.ec2.internal\" not found" Jul 07 18:22:57.291384 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.291295 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-108.ec2.internal"] Jul 07 18:22:57.291497 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.291435 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 18:22:57.292524 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.292504 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasSufficientMemory" Jul 07 18:22:57.292611 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.292540 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 18:22:57.292611 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.292552 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasSufficientPID" Jul 07 18:22:57.293987 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.293974 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 18:22:57.294118 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.294090 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal" Jul 07 18:22:57.294171 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.294138 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 18:22:57.294783 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.294764 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasSufficientMemory" Jul 07 18:22:57.294860 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.294794 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 18:22:57.294860 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.294806 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasSufficientPID" Jul 07 18:22:57.294860 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.294766 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasSufficientMemory" Jul 07 18:22:57.294952 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.294872 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 18:22:57.294952 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.294883 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasSufficientPID" Jul 07 18:22:57.296066 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.296052 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.ec2.internal" Jul 07 18:22:57.296130 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.296079 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 18:22:57.296804 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.296792 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasSufficientMemory" Jul 07 18:22:57.296852 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.296817 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 18:22:57.296852 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.296833 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeHasSufficientPID" Jul 07 18:22:57.321994 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.321963 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/810b738218389b4c948b52d7adabf33c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal\" (UID: \"810b738218389b4c948b52d7adabf33c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal" Jul 07 18:22:57.321994 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.321997 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/810b738218389b4c948b52d7adabf33c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal\" (UID: \"810b738218389b4c948b52d7adabf33c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal" Jul 07 18:22:57.322207 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.322017 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2b9e46c04925be656b0777b331f09d9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-108.ec2.internal\" (UID: \"f2b9e46c04925be656b0777b331f09d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.ec2.internal" Jul 07 18:22:57.326209 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.326192 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-108.ec2.internal\" not found" node="ip-10-0-141-108.ec2.internal" Jul 07 18:22:57.330927 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.330906 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-108.ec2.internal\" not found" node="ip-10-0-141-108.ec2.internal" Jul 07 18:22:57.346962 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.346937 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-108.ec2.internal\" not found" Jul 07 18:22:57.423177 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.423141 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/810b738218389b4c948b52d7adabf33c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal\" (UID: \"810b738218389b4c948b52d7adabf33c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal" Jul 07 18:22:57.423359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.423182 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/810b738218389b4c948b52d7adabf33c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal\" (UID: \"810b738218389b4c948b52d7adabf33c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal" Jul 07 18:22:57.423359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.423203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2b9e46c04925be656b0777b331f09d9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-108.ec2.internal\" (UID: \"f2b9e46c04925be656b0777b331f09d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.ec2.internal" Jul 07 18:22:57.423359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.423230 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/810b738218389b4c948b52d7adabf33c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal\" (UID: \"810b738218389b4c948b52d7adabf33c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal" Jul 07 18:22:57.423359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.423257 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2b9e46c04925be656b0777b331f09d9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-108.ec2.internal\" (UID: \"f2b9e46c04925be656b0777b331f09d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.ec2.internal" Jul 07 18:22:57.423359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.423275 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/810b738218389b4c948b52d7adabf33c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal\" (UID: \"810b738218389b4c948b52d7adabf33c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal" Jul 07 18:22:57.447300 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.447209 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-108.ec2.internal\" not found" Jul 07 18:22:57.547971 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.547936 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-108.ec2.internal\" not found" Jul 07 18:22:57.629458 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.629410 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal" Jul 07 18:22:57.634550 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.634527 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.ec2.internal" Jul 07 18:22:57.648805 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.648764 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-108.ec2.internal\" not found" Jul 07 18:22:57.749470 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.749372 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-108.ec2.internal\" not found" Jul 07 18:22:57.849969 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.849925 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-108.ec2.internal\" not found" Jul 07 18:22:57.900704 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.900671 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 18:22:57.950234 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:57.950193 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-108.ec2.internal\" not found" Jul 07 18:22:57.955517 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.955496 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 18:22:57.955700 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.955666 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 18:22:57.955820 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:57.955694 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 18:22:58.020159 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:58.020086 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 18:22:58.030277 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:58.030237 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 18:17:57 +0000 UTC" deadline="2027-12-03 20:58:46.592030841 +0000 UTC" Jul 07 18:22:58.030277 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:58.030266 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12338h35m48.561768678s" Jul 07 18:22:58.118752 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:58.118714 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal" Jul 07 18:22:58.131749 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:58.131724 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 18:22:58.133910 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:58.133887 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.ec2.internal" Jul 07 18:22:58.143948 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:58.143931 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 18:22:58.157171 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:58.157122 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b9e46c04925be656b0777b331f09d9.slice/crio-87b0dd7877d12d607b36d580f99b6222b37cbaebd7d95de1cd3b9112f9f31ba9 WatchSource:0}: Error finding container 87b0dd7877d12d607b36d580f99b6222b37cbaebd7d95de1cd3b9112f9f31ba9: Status 404 returned error can't find the container with id 87b0dd7877d12d607b36d580f99b6222b37cbaebd7d95de1cd3b9112f9f31ba9 Jul 07 18:22:58.157718 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:22:58.157692 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810b738218389b4c948b52d7adabf33c.slice/crio-036cc7c8623b10fb55b33af8969bc747bbe5cab875c27039d074bdcf1b5cbc68 WatchSource:0}: Error finding container 036cc7c8623b10fb55b33af8969bc747bbe5cab875c27039d074bdcf1b5cbc68: Status 404 returned error can't find the container with id 036cc7c8623b10fb55b33af8969bc747bbe5cab875c27039d074bdcf1b5cbc68 Jul 07 18:22:58.162410 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:58.162391 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 18:22:58.195100 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:58.195042 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.ec2.internal" event={"ID":"f2b9e46c04925be656b0777b331f09d9","Type":"ContainerStarted","Data":"87b0dd7877d12d607b36d580f99b6222b37cbaebd7d95de1cd3b9112f9f31ba9"} Jul 07 18:22:58.196061 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:58.196033 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal" event={"ID":"810b738218389b4c948b52d7adabf33c","Type":"ContainerStarted","Data":"036cc7c8623b10fb55b33af8969bc747bbe5cab875c27039d074bdcf1b5cbc68"} Jul 07 18:22:58.961284 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:58.961251 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 18:22:58.999518 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:58.999476 2578 apiserver.go:52] "Watching apiserver" Jul 07 18:22:59.004910 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.004880 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 18:22:59.005279 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.005256 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8","openshift-cluster-node-tuning-operator/tuned-sjrgs","openshift-image-registry/node-ca-wlm6k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal","openshift-multus/multus-9m4cn","openshift-network-operator/iptables-alerter-fjzj7","kube-system/global-pull-secret-syncer-98qb4","openshift-dns/node-resolver-bnkff","openshift-multus/multus-additional-cni-plugins-wgd4k","openshift-multus/network-metrics-daemon-bzlxp","openshift-network-diagnostics/network-check-target-k92pn","openshift-ovn-kubernetes/ovnkube-node-xcq8x","kube-system/konnectivity-agent-66cjm","kube-system/kube-apiserver-proxy-ip-10-0-141-108.ec2.internal"] Jul 07 18:22:59.007054 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.007030 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:22:59.007146 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:59.007123 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzlxp" podUID="cde64ccc-edac-45b8-b4d3-b260d022fbf5" Jul 07 18:22:59.008379 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.008364 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-66cjm" Jul 07 18:22:59.009598 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.009573 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wlm6k" Jul 07 18:22:59.010571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.010539 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-glmcp\"" Jul 07 18:22:59.010687 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.010540 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 18:22:59.010748 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.010728 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 18:22:59.010901 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.010861 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bnkff" Jul 07 18:22:59.012268 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.011779 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 18:22:59.012268 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.011789 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 18:22:59.012268 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.012017 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.012268 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.012031 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vx5b6\"" Jul 07 18:22:59.012268 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.012133 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 18:22:59.012754 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.012738 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 18:22:59.013136 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.013120 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bhns6\"" Jul 07 18:22:59.013370 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.013353 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fjzj7" Jul 07 18:22:59.014521 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.014491 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 18:22:59.014521 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.014508 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 18:22:59.014521 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.014516 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-79t47\"" Jul 07 18:22:59.014728 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.014532 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 18:22:59.014728 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.014535 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 18:22:59.014937 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.014920 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 18:22:59.015630 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.015611 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.015956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.015931 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-64ktj\"" Jul 07 18:22:59.016291 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.016152 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 18:22:59.016291 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.016182 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 18:22:59.016291 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.016262 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 18:22:59.017562 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.017545 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:22:59.017642 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:59.017606 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-98qb4" podUID="e17ddfe4-73b3-45bd-b80f-ed992e7e1b71" Jul 07 18:22:59.018199 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.018178 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 18:22:59.018527 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.018509 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k4wpl\"" Jul 07 18:22:59.018613 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.018561 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 18:22:59.019014 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.018987 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.019092 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.019025 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 18:22:59.020403 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.020384 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.021461 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.021375 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 18:22:59.021461 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.021382 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 18:22:59.021675 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.021477 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qc96l\"" Jul 07 18:22:59.021884 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.021863 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:22:59.021984 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:59.021939 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k92pn" podUID="3f672f83-cb85-4c82-91fd-1d47b9c67d59" Jul 07 18:22:59.022398 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.022382 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 18:22:59.022657 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.022640 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 18:22:59.022843 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.022828 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 18:22:59.022909 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.022855 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9sk2k\"" Jul 07 18:22:59.024084 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.023521 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.026929 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.026905 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 18:22:59.027531 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.027514 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-chfwp\"" Jul 07 18:22:59.027645 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.027627 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 18:22:59.028313 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.028296 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 18:22:59.028502 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.028484 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 18:22:59.028753 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.028579 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 18:22:59.028753 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.028646 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 18:22:59.031915 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.031851 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/936682ff-fe11-4ce3-8d8c-98c7b909b439-socket-dir\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.032003 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.031926 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88949732-a999-49ba-b1ba-3f283bbd1c45-konnectivity-ca\") pod \"konnectivity-agent-66cjm\" (UID: \"88949732-a999-49ba-b1ba-3f283bbd1c45\") " pod="kube-system/konnectivity-agent-66cjm" Jul 07 18:22:59.032003 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.031953 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/507b32a9-8506-40c1-b212-519a305e8caa-tmp\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.032003 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.031976 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-multus-cni-dir\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.032153 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032001 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b9f5\" (UniqueName: \"kubernetes.io/projected/c132a249-8a84-44eb-9198-2aaa0cb451fc-kube-api-access-7b9f5\") pod \"iptables-alerter-fjzj7\" (UID: \"c132a249-8a84-44eb-9198-2aaa0cb451fc\") " pod="openshift-network-operator/iptables-alerter-fjzj7" Jul 07 18:22:59.032153 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032025 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-kubelet-config\") pod \"global-pull-secret-syncer-98qb4\" (UID: \"e17ddfe4-73b3-45bd-b80f-ed992e7e1b71\") " pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:22:59.032153 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032048 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/936682ff-fe11-4ce3-8d8c-98c7b909b439-registration-dir\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.032153 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032084 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f1e7507b-7367-4253-bb48-212f27b5f8ca-os-release\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.032153 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032112 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp4xr\" (UniqueName: \"kubernetes.io/projected/f1e7507b-7367-4253-bb48-212f27b5f8ca-kube-api-access-fp4xr\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.032153 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032137 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-etc-sysconfig\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.032457 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032160 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c132a249-8a84-44eb-9198-2aaa0cb451fc-host-slash\") pod \"iptables-alerter-fjzj7\" (UID: \"c132a249-8a84-44eb-9198-2aaa0cb451fc\") " pod="openshift-network-operator/iptables-alerter-fjzj7" Jul 07 18:22:59.032457 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032219 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-etc-sysctl-conf\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.032457 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032242 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-system-cni-dir\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.032457 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032309 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-multus-conf-dir\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.032457 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032359 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-etc-sysctl-d\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.032457 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-var-lib-kubelet\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.032457 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-cni-binary-copy\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.032457 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032436 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/936682ff-fe11-4ce3-8d8c-98c7b909b439-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.032807 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032461 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-host\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.032807 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032512 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88949732-a999-49ba-b1ba-3f283bbd1c45-agent-certs\") pod \"konnectivity-agent-66cjm\" (UID: \"88949732-a999-49ba-b1ba-3f283bbd1c45\") " pod="kube-system/konnectivity-agent-66cjm" Jul 07 18:22:59.032807 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-etc-systemd\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.032807 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032585 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69cnn\" (UniqueName: \"kubernetes.io/projected/507b32a9-8506-40c1-b212-519a305e8caa-kube-api-access-69cnn\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.032807 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032610 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs\") pod \"network-metrics-daemon-bzlxp\" (UID: \"cde64ccc-edac-45b8-b4d3-b260d022fbf5\") " pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:22:59.032807 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032631 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-multus-daemon-config\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.032807 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032651 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-host-var-lib-cni-bin\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.032807 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032677 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-host-var-lib-kubelet\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.032807 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032710 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd4x9\" (UniqueName: \"kubernetes.io/projected/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-kube-api-access-xd4x9\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.032807 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032740 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-sys\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.032807 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032771 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/936682ff-fe11-4ce3-8d8c-98c7b909b439-device-dir\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.032807 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032797 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-host-var-lib-cni-multus\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.033419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032821 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret\") pod \"global-pull-secret-syncer-98qb4\" (UID: \"e17ddfe4-73b3-45bd-b80f-ed992e7e1b71\") " pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:22:59.033419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032844 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf3379f6-bbce-4398-bb71-1dbe3ddc5059-tmp-dir\") pod \"node-resolver-bnkff\" (UID: \"bf3379f6-bbce-4398-bb71-1dbe3ddc5059\") " pod="openshift-dns/node-resolver-bnkff" Jul 07 18:22:59.033419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032908 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjvfn\" (UniqueName: \"kubernetes.io/projected/bf3379f6-bbce-4398-bb71-1dbe3ddc5059-kube-api-access-gjvfn\") pod \"node-resolver-bnkff\" (UID: \"bf3379f6-bbce-4398-bb71-1dbe3ddc5059\") " pod="openshift-dns/node-resolver-bnkff" Jul 07 18:22:59.033419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032926 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-multus-socket-dir-parent\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.033419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032950 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f1e7507b-7367-4253-bb48-212f27b5f8ca-cni-binary-copy\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.033419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032967 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-run\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.033419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.032988 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3f4cdc4-da50-4c22-8e51-55872deaf1c8-serviceca\") pod \"node-ca-wlm6k\" (UID: \"d3f4cdc4-da50-4c22-8e51-55872deaf1c8\") " pod="openshift-image-registry/node-ca-wlm6k" Jul 07 18:22:59.033419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033012 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c49f9\" (UniqueName: \"kubernetes.io/projected/d3f4cdc4-da50-4c22-8e51-55872deaf1c8-kube-api-access-c49f9\") pod \"node-ca-wlm6k\" (UID: \"d3f4cdc4-da50-4c22-8e51-55872deaf1c8\") " pod="openshift-image-registry/node-ca-wlm6k" Jul 07 18:22:59.033419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033033 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-os-release\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.033419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033054 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-dbus\") pod \"global-pull-secret-syncer-98qb4\" (UID: \"e17ddfe4-73b3-45bd-b80f-ed992e7e1b71\") " pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:22:59.033419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033079 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f1e7507b-7367-4253-bb48-212f27b5f8ca-cnibin\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.033419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033102 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-etc-kubernetes\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.033419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033120 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c132a249-8a84-44eb-9198-2aaa0cb451fc-iptables-alerter-script\") pod \"iptables-alerter-fjzj7\" (UID: \"c132a249-8a84-44eb-9198-2aaa0cb451fc\") " pod="openshift-network-operator/iptables-alerter-fjzj7" Jul 07 18:22:59.033419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033142 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-etc-modprobe-d\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.033419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033162 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-cnibin\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.033419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033178 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/936682ff-fe11-4ce3-8d8c-98c7b909b439-etc-selinux\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.034188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033212 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f1e7507b-7367-4253-bb48-212f27b5f8ca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.034188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033250 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-lib-modules\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.034188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bf3379f6-bbce-4398-bb71-1dbe3ddc5059-hosts-file\") pod \"node-resolver-bnkff\" (UID: \"bf3379f6-bbce-4398-bb71-1dbe3ddc5059\") " pod="openshift-dns/node-resolver-bnkff" Jul 07 18:22:59.034188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033333 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-host-run-k8s-cni-cncf-io\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.034188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033351 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-host-run-netns\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.034188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033365 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-etc-kubernetes\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.034188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033391 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f1e7507b-7367-4253-bb48-212f27b5f8ca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.034188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033404 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3f4cdc4-da50-4c22-8e51-55872deaf1c8-host\") pod \"node-ca-wlm6k\" (UID: \"d3f4cdc4-da50-4c22-8e51-55872deaf1c8\") " pod="openshift-image-registry/node-ca-wlm6k" Jul 07 18:22:59.034188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033422 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-hostroot\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.034188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033476 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/936682ff-fe11-4ce3-8d8c-98c7b909b439-sys-fs\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.034188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033495 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwpfz\" (UniqueName: \"kubernetes.io/projected/936682ff-fe11-4ce3-8d8c-98c7b909b439-kube-api-access-nwpfz\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.034188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033517 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1e7507b-7367-4253-bb48-212f27b5f8ca-system-cni-dir\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.034188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033539 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f1e7507b-7367-4253-bb48-212f27b5f8ca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.034188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033561 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/507b32a9-8506-40c1-b212-519a305e8caa-etc-tuned\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.034188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033583 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vklwt\" (UniqueName: \"kubernetes.io/projected/cde64ccc-edac-45b8-b4d3-b260d022fbf5-kube-api-access-vklwt\") pod \"network-metrics-daemon-bzlxp\" (UID: \"cde64ccc-edac-45b8-b4d3-b260d022fbf5\") " pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:22:59.034188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.033617 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-host-run-multus-certs\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.034966 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.034208 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 18:22:59.060387 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.060302 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pm4r5" Jul 07 18:22:59.067861 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.067797 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pm4r5" Jul 07 18:22:59.120975 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.120901 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 18:22:59.134686 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.134649 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3f4cdc4-da50-4c22-8e51-55872deaf1c8-host\") pod \"node-ca-wlm6k\" (UID: \"d3f4cdc4-da50-4c22-8e51-55872deaf1c8\") " pod="openshift-image-registry/node-ca-wlm6k" Jul 07 18:22:59.134866 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.134701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-hostroot\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.134866 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.134729 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/936682ff-fe11-4ce3-8d8c-98c7b909b439-sys-fs\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.134866 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.134723 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3f4cdc4-da50-4c22-8e51-55872deaf1c8-host\") pod \"node-ca-wlm6k\" (UID: \"d3f4cdc4-da50-4c22-8e51-55872deaf1c8\") " pod="openshift-image-registry/node-ca-wlm6k" Jul 07 18:22:59.134866 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.134757 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwpfz\" (UniqueName: \"kubernetes.io/projected/936682ff-fe11-4ce3-8d8c-98c7b909b439-kube-api-access-nwpfz\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.134866 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.134844 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/936682ff-fe11-4ce3-8d8c-98c7b909b439-sys-fs\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.134866 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.134808 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1e7507b-7367-4253-bb48-212f27b5f8ca-system-cni-dir\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.134866 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.134839 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-hostroot\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.135149 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.134884 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f1e7507b-7367-4253-bb48-212f27b5f8ca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.135149 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.134894 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1e7507b-7367-4253-bb48-212f27b5f8ca-system-cni-dir\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.135149 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.134993 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-kubelet\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.135149 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135020 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-node-log\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.135149 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135043 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/507b32a9-8506-40c1-b212-519a305e8caa-etc-tuned\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.135149 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135095 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vklwt\" (UniqueName: \"kubernetes.io/projected/cde64ccc-edac-45b8-b4d3-b260d022fbf5-kube-api-access-vklwt\") pod \"network-metrics-daemon-bzlxp\" (UID: \"cde64ccc-edac-45b8-b4d3-b260d022fbf5\") " pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:22:59.135149 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135125 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-host-run-multus-certs\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.135149 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135148 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/936682ff-fe11-4ce3-8d8c-98c7b909b439-socket-dir\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.135538 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135174 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-host-run-multus-certs\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.135538 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135171 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88949732-a999-49ba-b1ba-3f283bbd1c45-konnectivity-ca\") pod \"konnectivity-agent-66cjm\" (UID: \"88949732-a999-49ba-b1ba-3f283bbd1c45\") " pod="kube-system/konnectivity-agent-66cjm" Jul 07 18:22:59.135538 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135225 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-run-openvswitch\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.135538 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135252 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-run-ovn\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.135538 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135279 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv4qg\" (UniqueName: \"kubernetes.io/projected/1d688309-1ae2-40ac-9de5-3b37cf893695-kube-api-access-tv4qg\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.135538 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/507b32a9-8506-40c1-b212-519a305e8caa-tmp\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.135538 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135363 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-multus-cni-dir\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.135538 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b9f5\" (UniqueName: \"kubernetes.io/projected/c132a249-8a84-44eb-9198-2aaa0cb451fc-kube-api-access-7b9f5\") pod \"iptables-alerter-fjzj7\" (UID: \"c132a249-8a84-44eb-9198-2aaa0cb451fc\") " pod="openshift-network-operator/iptables-alerter-fjzj7" Jul 07 18:22:59.135538 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-kubelet-config\") pod \"global-pull-secret-syncer-98qb4\" (UID: \"e17ddfe4-73b3-45bd-b80f-ed992e7e1b71\") " pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:22:59.135538 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135443 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/936682ff-fe11-4ce3-8d8c-98c7b909b439-registration-dir\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.135538 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135467 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f1e7507b-7367-4253-bb48-212f27b5f8ca-os-release\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.135538 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135484 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/936682ff-fe11-4ce3-8d8c-98c7b909b439-socket-dir\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.135538 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135500 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp4xr\" (UniqueName: \"kubernetes.io/projected/f1e7507b-7367-4253-bb48-212f27b5f8ca-kube-api-access-fp4xr\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.135538 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135526 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-systemd-units\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.135538 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135542 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/936682ff-fe11-4ce3-8d8c-98c7b909b439-registration-dir\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.136139 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135551 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-etc-sysconfig\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.136139 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135574 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c132a249-8a84-44eb-9198-2aaa0cb451fc-host-slash\") pod \"iptables-alerter-fjzj7\" (UID: \"c132a249-8a84-44eb-9198-2aaa0cb451fc\") " pod="openshift-network-operator/iptables-alerter-fjzj7" Jul 07 18:22:59.136139 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135581 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f1e7507b-7367-4253-bb48-212f27b5f8ca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.136139 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135596 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-run-ovn-kubernetes\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.136139 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-cni-bin\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.136139 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135645 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-etc-sysctl-conf\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.136139 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135661 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c132a249-8a84-44eb-9198-2aaa0cb451fc-host-slash\") pod \"iptables-alerter-fjzj7\" (UID: \"c132a249-8a84-44eb-9198-2aaa0cb451fc\") " pod="openshift-network-operator/iptables-alerter-fjzj7" Jul 07 18:22:59.136139 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-system-cni-dir\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.136139 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135588 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-kubelet-config\") pod \"global-pull-secret-syncer-98qb4\" (UID: \"e17ddfe4-73b3-45bd-b80f-ed992e7e1b71\") " pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:22:59.136139 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135695 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-multus-conf-dir\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.136139 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135666 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f1e7507b-7367-4253-bb48-212f27b5f8ca-os-release\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.136139 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135706 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88949732-a999-49ba-b1ba-3f283bbd1c45-konnectivity-ca\") pod \"konnectivity-agent-66cjm\" (UID: \"88949732-a999-49ba-b1ba-3f283bbd1c45\") " pod="kube-system/konnectivity-agent-66cjm" Jul 07 18:22:59.136139 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135720 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-etc-sysctl-d\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.136139 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135722 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-etc-sysconfig\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.136139 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135746 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-var-lib-kubelet\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.136139 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135767 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-multus-conf-dir\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.136139 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135771 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-cni-binary-copy\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.136900 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135807 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-system-cni-dir\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.136900 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135824 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/936682ff-fe11-4ce3-8d8c-98c7b909b439-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.136900 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135834 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-multus-cni-dir\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.136900 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135849 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d688309-1ae2-40ac-9de5-3b37cf893695-ovnkube-config\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.136900 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135863 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-etc-sysctl-conf\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.136900 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135887 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-etc-sysctl-d\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.136900 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135891 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-host\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.136900 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135907 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/936682ff-fe11-4ce3-8d8c-98c7b909b439-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.136900 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135910 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-var-lib-kubelet\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.136900 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135935 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 18:22:59.136900 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135950 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-host\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.136900 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.135975 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d688309-1ae2-40ac-9de5-3b37cf893695-env-overrides\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.136900 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136008 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88949732-a999-49ba-b1ba-3f283bbd1c45-agent-certs\") pod \"konnectivity-agent-66cjm\" (UID: \"88949732-a999-49ba-b1ba-3f283bbd1c45\") " pod="kube-system/konnectivity-agent-66cjm" Jul 07 18:22:59.136900 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136035 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-etc-systemd\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.136900 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69cnn\" (UniqueName: \"kubernetes.io/projected/507b32a9-8506-40c1-b212-519a305e8caa-kube-api-access-69cnn\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.136900 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136086 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs\") pod \"network-metrics-daemon-bzlxp\" (UID: \"cde64ccc-edac-45b8-b4d3-b260d022fbf5\") " pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:22:59.136900 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136106 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-multus-daemon-config\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.136900 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136131 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.138019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136157 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-host-var-lib-cni-bin\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.138019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136182 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-host-var-lib-kubelet\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.138019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136207 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd4x9\" (UniqueName: \"kubernetes.io/projected/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-kube-api-access-xd4x9\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.138019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136231 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-slash\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.138019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136252 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-cni-netd\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.138019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-sys\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.138019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136301 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/936682ff-fe11-4ce3-8d8c-98c7b909b439-device-dir\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.138019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136336 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-cni-binary-copy\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.138019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136341 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-var-lib-openvswitch\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.138019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136402 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-host-var-lib-cni-multus\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.138019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136403 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-host-var-lib-cni-bin\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.138019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136465 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-host-var-lib-cni-multus\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.138019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136467 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret\") pod \"global-pull-secret-syncer-98qb4\" (UID: \"e17ddfe4-73b3-45bd-b80f-ed992e7e1b71\") " pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:22:59.138019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136499 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-etc-openvswitch\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.138019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136520 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-log-socket\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.138019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136526 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-sys\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.138019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf3379f6-bbce-4398-bb71-1dbe3ddc5059-tmp-dir\") pod \"node-resolver-bnkff\" (UID: \"bf3379f6-bbce-4398-bb71-1dbe3ddc5059\") " pod="openshift-dns/node-resolver-bnkff" Jul 07 18:22:59.138019 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:59.136558 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 18:22:59.138917 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:59.136596 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:22:59.138917 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136636 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjvfn\" (UniqueName: \"kubernetes.io/projected/bf3379f6-bbce-4398-bb71-1dbe3ddc5059-kube-api-access-gjvfn\") pod \"node-resolver-bnkff\" (UID: \"bf3379f6-bbce-4398-bb71-1dbe3ddc5059\") " pod="openshift-dns/node-resolver-bnkff" Jul 07 18:22:59.138917 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136662 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-multus-socket-dir-parent\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.138917 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136720 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-multus-socket-dir-parent\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.138917 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136721 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-multus-daemon-config\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.138917 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:59.136734 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret podName:e17ddfe4-73b3-45bd-b80f-ed992e7e1b71 nodeName:}" failed. No retries permitted until 2026-07-07 18:22:59.636679653 +0000 UTC m=+3.020473031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret") pod "global-pull-secret-syncer-98qb4" (UID: "e17ddfe4-73b3-45bd-b80f-ed992e7e1b71") : object "kube-system"/"original-pull-secret" not registered Jul 07 18:22:59.138917 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136794 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-host-var-lib-kubelet\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.138917 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136763 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/936682ff-fe11-4ce3-8d8c-98c7b909b439-device-dir\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.138917 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136800 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f1e7507b-7367-4253-bb48-212f27b5f8ca-cni-binary-copy\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.138917 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136831 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-etc-systemd\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.138917 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136848 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-run-netns\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.138917 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136874 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-run\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.138917 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:59.136896 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs podName:cde64ccc-edac-45b8-b4d3-b260d022fbf5 nodeName:}" failed. No retries permitted until 2026-07-07 18:22:59.636879888 +0000 UTC m=+3.020673263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs") pod "network-metrics-daemon-bzlxp" (UID: "cde64ccc-edac-45b8-b4d3-b260d022fbf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:22:59.138917 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136915 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-run\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.138917 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136922 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3f4cdc4-da50-4c22-8e51-55872deaf1c8-serviceca\") pod \"node-ca-wlm6k\" (UID: \"d3f4cdc4-da50-4c22-8e51-55872deaf1c8\") " pod="openshift-image-registry/node-ca-wlm6k" Jul 07 18:22:59.138917 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c49f9\" (UniqueName: \"kubernetes.io/projected/d3f4cdc4-da50-4c22-8e51-55872deaf1c8-kube-api-access-c49f9\") pod \"node-ca-wlm6k\" (UID: \"d3f4cdc4-da50-4c22-8e51-55872deaf1c8\") " pod="openshift-image-registry/node-ca-wlm6k" Jul 07 18:22:59.139959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-os-release\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.139959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.136994 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-dbus\") pod \"global-pull-secret-syncer-98qb4\" (UID: \"e17ddfe4-73b3-45bd-b80f-ed992e7e1b71\") " pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:22:59.139959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137013 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f1e7507b-7367-4253-bb48-212f27b5f8ca-cnibin\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.139959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137063 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d688309-1ae2-40ac-9de5-3b37cf893695-ovnkube-script-lib\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.139959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137086 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-etc-kubernetes\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.139959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137110 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c132a249-8a84-44eb-9198-2aaa0cb451fc-iptables-alerter-script\") pod \"iptables-alerter-fjzj7\" (UID: \"c132a249-8a84-44eb-9198-2aaa0cb451fc\") " pod="openshift-network-operator/iptables-alerter-fjzj7" Jul 07 18:22:59.139959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137131 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kc7t\" (UniqueName: \"kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t\") pod \"network-check-target-k92pn\" (UID: \"3f672f83-cb85-4c82-91fd-1d47b9c67d59\") " pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:22:59.139959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137138 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf3379f6-bbce-4398-bb71-1dbe3ddc5059-tmp-dir\") pod \"node-resolver-bnkff\" (UID: \"bf3379f6-bbce-4398-bb71-1dbe3ddc5059\") " pod="openshift-dns/node-resolver-bnkff" Jul 07 18:22:59.139959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137156 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d688309-1ae2-40ac-9de5-3b37cf893695-ovn-node-metrics-cert\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.139959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137181 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-etc-modprobe-d\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.139959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137222 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-cnibin\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.139959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137251 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/936682ff-fe11-4ce3-8d8c-98c7b909b439-etc-selinux\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.139959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137272 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f1e7507b-7367-4253-bb48-212f27b5f8ca-cni-binary-copy\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.139959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137275 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-etc-modprobe-d\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.139959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137275 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f1e7507b-7367-4253-bb48-212f27b5f8ca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.139959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137348 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-lib-modules\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.139959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137341 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3f4cdc4-da50-4c22-8e51-55872deaf1c8-serviceca\") pod \"node-ca-wlm6k\" (UID: \"d3f4cdc4-da50-4c22-8e51-55872deaf1c8\") " pod="openshift-image-registry/node-ca-wlm6k" Jul 07 18:22:59.140664 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137390 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/936682ff-fe11-4ce3-8d8c-98c7b909b439-etc-selinux\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.140664 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137434 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f1e7507b-7367-4253-bb48-212f27b5f8ca-cnibin\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.140664 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137486 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-os-release\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.140664 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137548 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-etc-kubernetes\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.140664 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137554 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-dbus\") pod \"global-pull-secret-syncer-98qb4\" (UID: \"e17ddfe4-73b3-45bd-b80f-ed992e7e1b71\") " pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:22:59.140664 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137598 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bf3379f6-bbce-4398-bb71-1dbe3ddc5059-hosts-file\") pod \"node-resolver-bnkff\" (UID: \"bf3379f6-bbce-4398-bb71-1dbe3ddc5059\") " pod="openshift-dns/node-resolver-bnkff" Jul 07 18:22:59.140664 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137626 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-host-run-k8s-cni-cncf-io\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.140664 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137666 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-host-run-netns\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.140664 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137692 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-etc-kubernetes\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.140664 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137743 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f1e7507b-7367-4253-bb48-212f27b5f8ca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.140664 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137785 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/507b32a9-8506-40c1-b212-519a305e8caa-lib-modules\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.140664 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137787 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-run-systemd\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.140664 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137821 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-etc-kubernetes\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.140664 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137884 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-host-run-netns\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.140664 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137930 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bf3379f6-bbce-4398-bb71-1dbe3ddc5059-hosts-file\") pod \"node-resolver-bnkff\" (UID: \"bf3379f6-bbce-4398-bb71-1dbe3ddc5059\") " pod="openshift-dns/node-resolver-bnkff" Jul 07 18:22:59.140664 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.137948 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f1e7507b-7367-4253-bb48-212f27b5f8ca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.140664 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.138009 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-host-run-k8s-cni-cncf-io\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.140664 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.138024 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-cnibin\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.141274 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.138081 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c132a249-8a84-44eb-9198-2aaa0cb451fc-iptables-alerter-script\") pod \"iptables-alerter-fjzj7\" (UID: \"c132a249-8a84-44eb-9198-2aaa0cb451fc\") " pod="openshift-network-operator/iptables-alerter-fjzj7" Jul 07 18:22:59.141274 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.138476 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f1e7507b-7367-4253-bb48-212f27b5f8ca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.141274 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.140136 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88949732-a999-49ba-b1ba-3f283bbd1c45-agent-certs\") pod \"konnectivity-agent-66cjm\" (UID: \"88949732-a999-49ba-b1ba-3f283bbd1c45\") " pod="kube-system/konnectivity-agent-66cjm" Jul 07 18:22:59.141274 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.140488 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/507b32a9-8506-40c1-b212-519a305e8caa-tmp\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.141274 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.140538 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/507b32a9-8506-40c1-b212-519a305e8caa-etc-tuned\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.153892 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.153567 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vklwt\" (UniqueName: \"kubernetes.io/projected/cde64ccc-edac-45b8-b4d3-b260d022fbf5-kube-api-access-vklwt\") pod \"network-metrics-daemon-bzlxp\" (UID: \"cde64ccc-edac-45b8-b4d3-b260d022fbf5\") " pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:22:59.153892 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.153834 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c49f9\" (UniqueName: \"kubernetes.io/projected/d3f4cdc4-da50-4c22-8e51-55872deaf1c8-kube-api-access-c49f9\") pod \"node-ca-wlm6k\" (UID: \"d3f4cdc4-da50-4c22-8e51-55872deaf1c8\") " pod="openshift-image-registry/node-ca-wlm6k" Jul 07 18:22:59.154847 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.154819 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd4x9\" (UniqueName: \"kubernetes.io/projected/c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448-kube-api-access-xd4x9\") pod \"multus-9m4cn\" (UID: \"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448\") " pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.155269 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.155246 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjvfn\" (UniqueName: \"kubernetes.io/projected/bf3379f6-bbce-4398-bb71-1dbe3ddc5059-kube-api-access-gjvfn\") pod \"node-resolver-bnkff\" (UID: \"bf3379f6-bbce-4398-bb71-1dbe3ddc5059\") " pod="openshift-dns/node-resolver-bnkff" Jul 07 18:22:59.155382 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.155363 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69cnn\" (UniqueName: \"kubernetes.io/projected/507b32a9-8506-40c1-b212-519a305e8caa-kube-api-access-69cnn\") pod \"tuned-sjrgs\" (UID: \"507b32a9-8506-40c1-b212-519a305e8caa\") " pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.155478 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.155457 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b9f5\" (UniqueName: \"kubernetes.io/projected/c132a249-8a84-44eb-9198-2aaa0cb451fc-kube-api-access-7b9f5\") pod \"iptables-alerter-fjzj7\" (UID: \"c132a249-8a84-44eb-9198-2aaa0cb451fc\") " pod="openshift-network-operator/iptables-alerter-fjzj7" Jul 07 18:22:59.156600 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.156528 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwpfz\" (UniqueName: \"kubernetes.io/projected/936682ff-fe11-4ce3-8d8c-98c7b909b439-kube-api-access-nwpfz\") pod \"aws-ebs-csi-driver-node-x8vn8\" (UID: \"936682ff-fe11-4ce3-8d8c-98c7b909b439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.157118 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.157099 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp4xr\" (UniqueName: \"kubernetes.io/projected/f1e7507b-7367-4253-bb48-212f27b5f8ca-kube-api-access-fp4xr\") pod \"multus-additional-cni-plugins-wgd4k\" (UID: \"f1e7507b-7367-4253-bb48-212f27b5f8ca\") " pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.238510 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.238425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d688309-1ae2-40ac-9de5-3b37cf893695-env-overrides\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.238510 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.238481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.238906 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.238867 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-slash\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.238992 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.238926 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-cni-netd\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.238992 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.238957 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-var-lib-openvswitch\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239090 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.238998 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-etc-openvswitch\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239090 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239023 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-log-socket\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239090 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239051 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-run-netns\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239090 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239078 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d688309-1ae2-40ac-9de5-3b37cf893695-ovnkube-script-lib\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239283 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239106 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kc7t\" (UniqueName: \"kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t\") pod \"network-check-target-k92pn\" (UID: \"3f672f83-cb85-4c82-91fd-1d47b9c67d59\") " pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:22:59.239283 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239105 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-cni-netd\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239283 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239127 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-run-netns\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239283 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239072 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d688309-1ae2-40ac-9de5-3b37cf893695-env-overrides\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239283 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239126 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239283 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239125 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-etc-openvswitch\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239283 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239130 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d688309-1ae2-40ac-9de5-3b37cf893695-ovn-node-metrics-cert\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239283 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239214 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-run-systemd\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239283 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239170 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-var-lib-openvswitch\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239283 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239274 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-kubelet\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239285 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-slash\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239299 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-run-systemd\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239297 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-node-log\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239354 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-kubelet\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239357 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-log-socket\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239356 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-run-openvswitch\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239390 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-node-log\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239389 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-run-ovn\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239419 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-run-ovn\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239419 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4qg\" (UniqueName: \"kubernetes.io/projected/1d688309-1ae2-40ac-9de5-3b37cf893695-kube-api-access-tv4qg\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-systemd-units\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239467 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-run-openvswitch\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-run-ovn-kubernetes\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239506 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-cni-bin\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239509 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-systemd-units\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239552 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-cni-bin\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239562 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d688309-1ae2-40ac-9de5-3b37cf893695-host-run-ovn-kubernetes\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.239760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239607 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d688309-1ae2-40ac-9de5-3b37cf893695-ovnkube-config\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.240506 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.239797 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d688309-1ae2-40ac-9de5-3b37cf893695-ovnkube-script-lib\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.240506 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.240047 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d688309-1ae2-40ac-9de5-3b37cf893695-ovnkube-config\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.242147 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.242119 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d688309-1ae2-40ac-9de5-3b37cf893695-ovn-node-metrics-cert\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.253023 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:59.252992 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 18:22:59.253023 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:59.253020 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 18:22:59.253023 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:59.253031 2578 projected.go:196] Error preparing data for projected volume kube-api-access-6kc7t for pod openshift-network-diagnostics/network-check-target-k92pn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:22:59.253299 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:59.253121 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t podName:3f672f83-cb85-4c82-91fd-1d47b9c67d59 nodeName:}" failed. No retries permitted until 2026-07-07 18:22:59.753096543 +0000 UTC m=+3.136889917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6kc7t" (UniqueName: "kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t") pod "network-check-target-k92pn" (UID: "3f672f83-cb85-4c82-91fd-1d47b9c67d59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:22:59.255802 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.255773 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv4qg\" (UniqueName: \"kubernetes.io/projected/1d688309-1ae2-40ac-9de5-3b37cf893695-kube-api-access-tv4qg\") pod \"ovnkube-node-xcq8x\" (UID: \"1d688309-1ae2-40ac-9de5-3b37cf893695\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.272446 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.272411 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 18:22:59.321251 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.321212 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-66cjm" Jul 07 18:22:59.329365 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.329333 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wlm6k" Jul 07 18:22:59.349568 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.349528 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bnkff" Jul 07 18:22:59.354522 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.354500 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9m4cn" Jul 07 18:22:59.361268 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.361241 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fjzj7" Jul 07 18:22:59.370054 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.370020 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wgd4k" Jul 07 18:22:59.377139 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.377114 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" Jul 07 18:22:59.385115 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.385090 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" Jul 07 18:22:59.392453 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.392425 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:22:59.642643 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.642545 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs\") pod \"network-metrics-daemon-bzlxp\" (UID: \"cde64ccc-edac-45b8-b4d3-b260d022fbf5\") " pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:22:59.642643 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.642598 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret\") pod \"global-pull-secret-syncer-98qb4\" (UID: \"e17ddfe4-73b3-45bd-b80f-ed992e7e1b71\") " pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:22:59.642874 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:59.642719 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 18:22:59.642874 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:59.642800 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret podName:e17ddfe4-73b3-45bd-b80f-ed992e7e1b71 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:00.64278063 +0000 UTC m=+4.026574018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret") pod "global-pull-secret-syncer-98qb4" (UID: "e17ddfe4-73b3-45bd-b80f-ed992e7e1b71") : object "kube-system"/"original-pull-secret" not registered Jul 07 18:22:59.642874 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:59.642719 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:22:59.643025 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:59.642880 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs podName:cde64ccc-edac-45b8-b4d3-b260d022fbf5 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:00.642863271 +0000 UTC m=+4.026656648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs") pod "network-metrics-daemon-bzlxp" (UID: "cde64ccc-edac-45b8-b4d3-b260d022fbf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:22:59.843808 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:22:59.843762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kc7t\" (UniqueName: \"kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t\") pod \"network-check-target-k92pn\" (UID: \"3f672f83-cb85-4c82-91fd-1d47b9c67d59\") " pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:22:59.843997 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:59.843950 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 18:22:59.843997 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:59.843981 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 18:22:59.843997 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:59.843995 2578 projected.go:196] Error preparing data for projected volume kube-api-access-6kc7t for pod openshift-network-diagnostics/network-check-target-k92pn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:22:59.844122 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:22:59.844055 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t podName:3f672f83-cb85-4c82-91fd-1d47b9c67d59 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:00.844040873 +0000 UTC m=+4.227834251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kc7t" (UniqueName: "kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t") pod "network-check-target-k92pn" (UID: "3f672f83-cb85-4c82-91fd-1d47b9c67d59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:23:00.001761 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:23:00.001729 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88949732_a999_49ba_b1ba_3f283bbd1c45.slice/crio-7280fb9a73cee61330ceaed158eacb707e76d28b3fb75699878d48fd109e6b2f WatchSource:0}: Error finding container 7280fb9a73cee61330ceaed158eacb707e76d28b3fb75699878d48fd109e6b2f: Status 404 returned error can't find the container with id 7280fb9a73cee61330ceaed158eacb707e76d28b3fb75699878d48fd109e6b2f Jul 07 18:23:00.005146 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:23:00.005035 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3379f6_bbce_4398_bb71_1dbe3ddc5059.slice/crio-ba2e36f2ba3e424fa88a2c71f0d5343732e973fe5b6d67e284bc6e77185fd910 WatchSource:0}: Error finding container ba2e36f2ba3e424fa88a2c71f0d5343732e973fe5b6d67e284bc6e77185fd910: Status 404 returned error can't find the container with id ba2e36f2ba3e424fa88a2c71f0d5343732e973fe5b6d67e284bc6e77185fd910 Jul 07 18:23:00.007294 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:23:00.007269 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f4cdc4_da50_4c22_8e51_55872deaf1c8.slice/crio-537ca1fb245a4bbbcc774eeec19b82c51617aa667aa5fa4e7250d6baeea5da41 WatchSource:0}: Error finding container 537ca1fb245a4bbbcc774eeec19b82c51617aa667aa5fa4e7250d6baeea5da41: Status 404 returned error can't find the container with id 537ca1fb245a4bbbcc774eeec19b82c51617aa667aa5fa4e7250d6baeea5da41 Jul 07 18:23:00.008261 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:23:00.008233 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc132a249_8a84_44eb_9198_2aaa0cb451fc.slice/crio-c4eb3031168e24d8c9a8d1044409051246ceda6b62f687483d15f7837014ae3d WatchSource:0}: Error finding container c4eb3031168e24d8c9a8d1044409051246ceda6b62f687483d15f7837014ae3d: Status 404 returned error can't find the container with id c4eb3031168e24d8c9a8d1044409051246ceda6b62f687483d15f7837014ae3d Jul 07 18:23:00.009513 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:23:00.009494 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936682ff_fe11_4ce3_8d8c_98c7b909b439.slice/crio-028c0c550180ca0a4bf1f96044b45aba53acb75b7bfccd7b80a9b53e92f11081 WatchSource:0}: Error finding container 028c0c550180ca0a4bf1f96044b45aba53acb75b7bfccd7b80a9b53e92f11081: Status 404 returned error can't find the container with id 028c0c550180ca0a4bf1f96044b45aba53acb75b7bfccd7b80a9b53e92f11081 Jul 07 18:23:00.010098 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:23:00.009978 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c36400_3fba_4f9a_9cf3_0d8cf8ebe448.slice/crio-1bf80dc841fb5b72094913613ea55a4b55727ea34d3fc58f7663a61b8298ad6b WatchSource:0}: Error finding container 1bf80dc841fb5b72094913613ea55a4b55727ea34d3fc58f7663a61b8298ad6b: Status 404 returned error can't find the container with id 1bf80dc841fb5b72094913613ea55a4b55727ea34d3fc58f7663a61b8298ad6b Jul 07 18:23:00.012173 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:23:00.011942 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d688309_1ae2_40ac_9de5_3b37cf893695.slice/crio-1886200b2624595cb3bcf8ed3517505be4f4c3ef9b4684880e7e973e2866e3a3 WatchSource:0}: Error finding container 1886200b2624595cb3bcf8ed3517505be4f4c3ef9b4684880e7e973e2866e3a3: Status 404 returned error can't find the container with id 1886200b2624595cb3bcf8ed3517505be4f4c3ef9b4684880e7e973e2866e3a3 Jul 07 18:23:00.012466 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:23:00.012387 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e7507b_7367_4253_bb48_212f27b5f8ca.slice/crio-5aa2499dcee45694aaf13bd278e81017f5893cfc9fc18cdf09fe56d0cca8cf04 WatchSource:0}: Error finding container 5aa2499dcee45694aaf13bd278e81017f5893cfc9fc18cdf09fe56d0cca8cf04: Status 404 returned error can't find the container with id 5aa2499dcee45694aaf13bd278e81017f5893cfc9fc18cdf09fe56d0cca8cf04 Jul 07 18:23:00.013521 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:23:00.013500 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507b32a9_8506_40c1_b212_519a305e8caa.slice/crio-0b2a8fdf45d89d960f235c43d5038d94ea08ad17239330f4962976231f33f5e3 WatchSource:0}: Error finding container 0b2a8fdf45d89d960f235c43d5038d94ea08ad17239330f4962976231f33f5e3: Status 404 returned error can't find the container with id 0b2a8fdf45d89d960f235c43d5038d94ea08ad17239330f4962976231f33f5e3 Jul 07 18:23:00.071555 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:00.071346 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 18:17:59 +0000 UTC" deadline="2028-03-12 12:13:10.447110108 +0000 UTC" Jul 07 18:23:00.071555 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:00.071554 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14729h50m10.375561735s" Jul 07 18:23:00.202383 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:00.202343 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wgd4k" event={"ID":"f1e7507b-7367-4253-bb48-212f27b5f8ca","Type":"ContainerStarted","Data":"5aa2499dcee45694aaf13bd278e81017f5893cfc9fc18cdf09fe56d0cca8cf04"} Jul 07 18:23:00.203649 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:00.203619 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" event={"ID":"1d688309-1ae2-40ac-9de5-3b37cf893695","Type":"ContainerStarted","Data":"1886200b2624595cb3bcf8ed3517505be4f4c3ef9b4684880e7e973e2866e3a3"} Jul 07 18:23:00.204845 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:00.204819 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9m4cn" event={"ID":"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448","Type":"ContainerStarted","Data":"1bf80dc841fb5b72094913613ea55a4b55727ea34d3fc58f7663a61b8298ad6b"} Jul 07 18:23:00.206029 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:00.205988 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" event={"ID":"936682ff-fe11-4ce3-8d8c-98c7b909b439","Type":"ContainerStarted","Data":"028c0c550180ca0a4bf1f96044b45aba53acb75b7bfccd7b80a9b53e92f11081"} Jul 07 18:23:00.207218 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:00.207182 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wlm6k" event={"ID":"d3f4cdc4-da50-4c22-8e51-55872deaf1c8","Type":"ContainerStarted","Data":"537ca1fb245a4bbbcc774eeec19b82c51617aa667aa5fa4e7250d6baeea5da41"} Jul 07 18:23:00.208156 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:00.208135 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bnkff" event={"ID":"bf3379f6-bbce-4398-bb71-1dbe3ddc5059","Type":"ContainerStarted","Data":"ba2e36f2ba3e424fa88a2c71f0d5343732e973fe5b6d67e284bc6e77185fd910"} Jul 07 18:23:00.210371 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:00.210349 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-66cjm" event={"ID":"88949732-a999-49ba-b1ba-3f283bbd1c45","Type":"ContainerStarted","Data":"7280fb9a73cee61330ceaed158eacb707e76d28b3fb75699878d48fd109e6b2f"} Jul 07 18:23:00.213069 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:00.213046 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.ec2.internal" event={"ID":"f2b9e46c04925be656b0777b331f09d9","Type":"ContainerStarted","Data":"1dbf780d72da0d6e7aa2b4c123d9318adecd1c77bbeb09595255d8f39e28ea93"} Jul 07 18:23:00.214175 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:00.214146 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fjzj7" event={"ID":"c132a249-8a84-44eb-9198-2aaa0cb451fc","Type":"ContainerStarted","Data":"c4eb3031168e24d8c9a8d1044409051246ceda6b62f687483d15f7837014ae3d"} Jul 07 18:23:00.215302 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:00.215280 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" event={"ID":"507b32a9-8506-40c1-b212-519a305e8caa","Type":"ContainerStarted","Data":"0b2a8fdf45d89d960f235c43d5038d94ea08ad17239330f4962976231f33f5e3"} Jul 07 18:23:00.229146 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:00.229098 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.ec2.internal" podStartSLOduration=2.229082024 podStartE2EDuration="2.229082024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:00.228825531 +0000 UTC m=+3.612618922" watchObservedRunningTime="2026-07-07 18:23:00.229082024 +0000 UTC m=+3.612875455" Jul 07 18:23:00.443348 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:00.443300 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 18:23:00.653308 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:00.653220 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs\") pod \"network-metrics-daemon-bzlxp\" (UID: \"cde64ccc-edac-45b8-b4d3-b260d022fbf5\") " pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:00.653308 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:00.653276 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret\") pod \"global-pull-secret-syncer-98qb4\" (UID: \"e17ddfe4-73b3-45bd-b80f-ed992e7e1b71\") " pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:00.653559 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:00.653432 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 18:23:00.653559 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:00.653494 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret podName:e17ddfe4-73b3-45bd-b80f-ed992e7e1b71 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:02.653476244 +0000 UTC m=+6.037269632 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret") pod "global-pull-secret-syncer-98qb4" (UID: "e17ddfe4-73b3-45bd-b80f-ed992e7e1b71") : object "kube-system"/"original-pull-secret" not registered Jul 07 18:23:00.654050 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:00.653891 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:23:00.654050 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:00.653942 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs podName:cde64ccc-edac-45b8-b4d3-b260d022fbf5 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:02.653925828 +0000 UTC m=+6.037719204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs") pod "network-metrics-daemon-bzlxp" (UID: "cde64ccc-edac-45b8-b4d3-b260d022fbf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:23:00.854656 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:00.854607 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kc7t\" (UniqueName: \"kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t\") pod \"network-check-target-k92pn\" (UID: \"3f672f83-cb85-4c82-91fd-1d47b9c67d59\") " pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:00.854849 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:00.854791 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 18:23:00.854849 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:00.854815 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 18:23:00.854849 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:00.854828 2578 projected.go:196] Error preparing data for projected volume kube-api-access-6kc7t for pod openshift-network-diagnostics/network-check-target-k92pn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:23:00.854998 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:00.854891 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t podName:3f672f83-cb85-4c82-91fd-1d47b9c67d59 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:02.854870574 +0000 UTC m=+6.238663964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kc7t" (UniqueName: "kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t") pod "network-check-target-k92pn" (UID: "3f672f83-cb85-4c82-91fd-1d47b9c67d59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:23:01.073026 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:01.072400 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 18:17:59 +0000 UTC" deadline="2028-03-19 22:36:00.773657282 +0000 UTC" Jul 07 18:23:01.073026 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:01.072464 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14908h12m59.70119682s" Jul 07 18:23:01.192180 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:01.192147 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:01.192373 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:01.192284 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k92pn" podUID="3f672f83-cb85-4c82-91fd-1d47b9c67d59" Jul 07 18:23:01.192373 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:01.192147 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:01.192373 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:01.192366 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:01.192530 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:01.192453 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzlxp" podUID="cde64ccc-edac-45b8-b4d3-b260d022fbf5" Jul 07 18:23:01.192693 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:01.192596 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-98qb4" podUID="e17ddfe4-73b3-45bd-b80f-ed992e7e1b71" Jul 07 18:23:01.238936 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:01.238558 2578 generic.go:356] "Generic (PLEG): container finished" podID="810b738218389b4c948b52d7adabf33c" containerID="1a2f8f8f679d2342f5dc2e49213a90892b9c84deb0441697c1ea8c50af2908d2" exitCode=0 Jul 07 18:23:01.238936 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:01.238701 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal" event={"ID":"810b738218389b4c948b52d7adabf33c","Type":"ContainerDied","Data":"1a2f8f8f679d2342f5dc2e49213a90892b9c84deb0441697c1ea8c50af2908d2"} Jul 07 18:23:02.250882 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:02.250841 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal" event={"ID":"810b738218389b4c948b52d7adabf33c","Type":"ContainerStarted","Data":"02186f253944f5cc54d09883caa108c6f0173225b2a43efd02f5d0c31fed2e84"} Jul 07 18:23:02.672369 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:02.672265 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret\") pod \"global-pull-secret-syncer-98qb4\" (UID: \"e17ddfe4-73b3-45bd-b80f-ed992e7e1b71\") " pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:02.672539 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:02.672390 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs\") pod \"network-metrics-daemon-bzlxp\" (UID: \"cde64ccc-edac-45b8-b4d3-b260d022fbf5\") " pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:02.672596 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:02.672540 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:23:02.672644 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:02.672604 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs podName:cde64ccc-edac-45b8-b4d3-b260d022fbf5 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:06.67258576 +0000 UTC m=+10.056379134 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs") pod "network-metrics-daemon-bzlxp" (UID: "cde64ccc-edac-45b8-b4d3-b260d022fbf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:23:02.672870 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:02.672847 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 18:23:02.672975 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:02.672915 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret podName:e17ddfe4-73b3-45bd-b80f-ed992e7e1b71 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:06.672897356 +0000 UTC m=+10.056690739 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret") pod "global-pull-secret-syncer-98qb4" (UID: "e17ddfe4-73b3-45bd-b80f-ed992e7e1b71") : object "kube-system"/"original-pull-secret" not registered Jul 07 18:23:02.873950 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:02.873909 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kc7t\" (UniqueName: \"kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t\") pod \"network-check-target-k92pn\" (UID: \"3f672f83-cb85-4c82-91fd-1d47b9c67d59\") " pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:02.874243 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:02.874166 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 18:23:02.874243 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:02.874190 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 18:23:02.874243 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:02.874203 2578 projected.go:196] Error preparing data for projected volume kube-api-access-6kc7t for pod openshift-network-diagnostics/network-check-target-k92pn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:23:02.874450 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:02.874265 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t podName:3f672f83-cb85-4c82-91fd-1d47b9c67d59 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:06.874246109 +0000 UTC m=+10.258039507 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kc7t" (UniqueName: "kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t") pod "network-check-target-k92pn" (UID: "3f672f83-cb85-4c82-91fd-1d47b9c67d59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:23:03.192454 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:03.191937 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:03.192454 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:03.192080 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k92pn" podUID="3f672f83-cb85-4c82-91fd-1d47b9c67d59" Jul 07 18:23:03.192454 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:03.192384 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:03.192754 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:03.192504 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzlxp" podUID="cde64ccc-edac-45b8-b4d3-b260d022fbf5" Jul 07 18:23:03.192754 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:03.192574 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:03.192754 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:03.192647 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-98qb4" podUID="e17ddfe4-73b3-45bd-b80f-ed992e7e1b71" Jul 07 18:23:05.191974 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:05.191705 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:05.191974 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:05.191705 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:05.191974 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:05.191829 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-98qb4" podUID="e17ddfe4-73b3-45bd-b80f-ed992e7e1b71" Jul 07 18:23:05.191974 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:05.191861 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:05.191974 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:05.191950 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k92pn" podUID="3f672f83-cb85-4c82-91fd-1d47b9c67d59" Jul 07 18:23:05.192642 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:05.192048 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzlxp" podUID="cde64ccc-edac-45b8-b4d3-b260d022fbf5" Jul 07 18:23:06.706077 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:06.706028 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs\") pod \"network-metrics-daemon-bzlxp\" (UID: \"cde64ccc-edac-45b8-b4d3-b260d022fbf5\") " pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:06.706612 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:06.706098 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret\") pod \"global-pull-secret-syncer-98qb4\" (UID: \"e17ddfe4-73b3-45bd-b80f-ed992e7e1b71\") " pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:06.706612 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:06.706196 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:23:06.706612 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:06.706269 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 18:23:06.706612 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:06.706275 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs podName:cde64ccc-edac-45b8-b4d3-b260d022fbf5 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:14.706253373 +0000 UTC m=+18.090046769 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs") pod "network-metrics-daemon-bzlxp" (UID: "cde64ccc-edac-45b8-b4d3-b260d022fbf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:23:06.706612 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:06.706336 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret podName:e17ddfe4-73b3-45bd-b80f-ed992e7e1b71 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:14.706305044 +0000 UTC m=+18.090098423 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret") pod "global-pull-secret-syncer-98qb4" (UID: "e17ddfe4-73b3-45bd-b80f-ed992e7e1b71") : object "kube-system"/"original-pull-secret" not registered Jul 07 18:23:06.907608 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:06.907565 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kc7t\" (UniqueName: \"kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t\") pod \"network-check-target-k92pn\" (UID: \"3f672f83-cb85-4c82-91fd-1d47b9c67d59\") " pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:06.907806 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:06.907750 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 18:23:06.907806 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:06.907774 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 18:23:06.907806 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:06.907786 2578 projected.go:196] Error preparing data for projected volume kube-api-access-6kc7t for pod openshift-network-diagnostics/network-check-target-k92pn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:23:06.907937 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:06.907854 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t podName:3f672f83-cb85-4c82-91fd-1d47b9c67d59 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:14.907834924 +0000 UTC m=+18.291628320 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kc7t" (UniqueName: "kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t") pod "network-check-target-k92pn" (UID: "3f672f83-cb85-4c82-91fd-1d47b9c67d59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:23:07.192402 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:07.192307 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:07.192402 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:07.192365 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:07.192607 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:07.192469 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzlxp" podUID="cde64ccc-edac-45b8-b4d3-b260d022fbf5" Jul 07 18:23:07.192607 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:07.192496 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-98qb4" podUID="e17ddfe4-73b3-45bd-b80f-ed992e7e1b71" Jul 07 18:23:07.192607 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:07.192545 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:07.192742 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:07.192657 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k92pn" podUID="3f672f83-cb85-4c82-91fd-1d47b9c67d59" Jul 07 18:23:09.191505 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:09.191465 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:09.191505 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:09.191464 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:09.192024 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:09.191685 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-98qb4" podUID="e17ddfe4-73b3-45bd-b80f-ed992e7e1b71" Jul 07 18:23:09.192024 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:09.191712 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:09.192024 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:09.191823 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzlxp" podUID="cde64ccc-edac-45b8-b4d3-b260d022fbf5" Jul 07 18:23:09.192024 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:09.191894 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k92pn" podUID="3f672f83-cb85-4c82-91fd-1d47b9c67d59" Jul 07 18:23:10.270786 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:10.270479 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" event={"ID":"507b32a9-8506-40c1-b212-519a305e8caa","Type":"ContainerStarted","Data":"fc83daaef57ed1b7ae962339b5fef5af1de4d92f61b4f9ec006e38b06d1638da"} Jul 07 18:23:10.272198 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:10.272167 2578 generic.go:356] "Generic (PLEG): container finished" podID="f1e7507b-7367-4253-bb48-212f27b5f8ca" containerID="8c90e48e5f9b2dd982838caa3d6d60a823c1381569e2adb318604d2f31cfad3e" exitCode=0 Jul 07 18:23:10.272362 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:10.272247 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wgd4k" event={"ID":"f1e7507b-7367-4253-bb48-212f27b5f8ca","Type":"ContainerDied","Data":"8c90e48e5f9b2dd982838caa3d6d60a823c1381569e2adb318604d2f31cfad3e"} Jul 07 18:23:10.274207 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:10.274176 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" event={"ID":"936682ff-fe11-4ce3-8d8c-98c7b909b439","Type":"ContainerStarted","Data":"e3aded5ff61ec766b54d82ed5591f2c66a3caac9badd274713aff977cd0afe17"} Jul 07 18:23:10.279543 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:10.279195 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wlm6k" event={"ID":"d3f4cdc4-da50-4c22-8e51-55872deaf1c8","Type":"ContainerStarted","Data":"d0a4da81c944798e625cc85b22fb7186e83004b65c875776e5d75386f2ec0325"} Jul 07 18:23:10.281735 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:10.281671 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bnkff" event={"ID":"bf3379f6-bbce-4398-bb71-1dbe3ddc5059","Type":"ContainerStarted","Data":"616583966b7f6ba269fae43fc6fb5fe14b89d1ddcaa429142b3b61a10bb2a9d1"} Jul 07 18:23:10.283516 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:10.283463 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-66cjm" event={"ID":"88949732-a999-49ba-b1ba-3f283bbd1c45","Type":"ContainerStarted","Data":"3dddca653f987560db487c0503ec1205354cf0a35498e502ce953f3d8f26f9b1"} Jul 07 18:23:10.296514 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:10.296449 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sjrgs" podStartSLOduration=3.625632337 podStartE2EDuration="13.29643404s" totalImagesPullingTime="9.670801703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:10.296388141 +0000 UTC m=+13.680181538" watchObservedRunningTime="2026-07-07 18:23:10.29643404 +0000 UTC m=+13.680227434" Jul 07 18:23:10.296693 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:10.296626 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.ec2.internal" podStartSLOduration=12.296616689 podStartE2EDuration="12.296616689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:02.286705487 +0000 UTC m=+5.670498884" watchObservedRunningTime="2026-07-07 18:23:10.296616689 +0000 UTC m=+13.680410084" Jul 07 18:23:10.315008 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:10.314953 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-66cjm" podStartSLOduration=3.6794251239999998 podStartE2EDuration="13.314938177s" totalImagesPullingTime="9.635513053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:10.314578148 +0000 UTC m=+13.698371554" watchObservedRunningTime="2026-07-07 18:23:10.314938177 +0000 UTC m=+13.698731573" Jul 07 18:23:10.331997 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:10.331935 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wlm6k" podStartSLOduration=3.6791466760000002 podStartE2EDuration="13.331918114s" totalImagesPullingTime="9.652771438s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:10.331411865 +0000 UTC m=+13.715205261" watchObservedRunningTime="2026-07-07 18:23:10.331918114 +0000 UTC m=+13.715711507" Jul 07 18:23:10.351693 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:10.351555 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bnkff" podStartSLOduration=3.700573426 podStartE2EDuration="13.351535455s" totalImagesPullingTime="9.650962029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:10.350936089 +0000 UTC m=+13.734729485" watchObservedRunningTime="2026-07-07 18:23:10.351535455 +0000 UTC m=+13.735328849" Jul 07 18:23:11.191958 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:11.191926 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:11.192120 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:11.191926 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:11.192120 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:11.192065 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzlxp" podUID="cde64ccc-edac-45b8-b4d3-b260d022fbf5" Jul 07 18:23:11.192120 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:11.191927 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:11.192285 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:11.192151 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k92pn" podUID="3f672f83-cb85-4c82-91fd-1d47b9c67d59" Jul 07 18:23:11.192285 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:11.192197 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-98qb4" podUID="e17ddfe4-73b3-45bd-b80f-ed992e7e1b71" Jul 07 18:23:11.287397 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:11.287345 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fjzj7" event={"ID":"c132a249-8a84-44eb-9198-2aaa0cb451fc","Type":"ContainerStarted","Data":"6e548ecec98f469f002db979e71cfedea3a7e82ab19b897ef08dfe884cdf3895"} Jul 07 18:23:11.304981 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:11.304918 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fjzj7" podStartSLOduration=4.656397707 podStartE2EDuration="14.30489481s" totalImagesPullingTime="9.648497103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:11.304356524 +0000 UTC m=+14.688149918" watchObservedRunningTime="2026-07-07 18:23:11.30489481 +0000 UTC m=+14.688688204" Jul 07 18:23:13.019018 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:13.018971 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-66cjm" Jul 07 18:23:13.019648 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:13.019627 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-66cjm" Jul 07 18:23:13.191659 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:13.191430 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:13.191659 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:13.191430 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:13.191659 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:13.191575 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:13.191659 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:13.191590 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzlxp" podUID="cde64ccc-edac-45b8-b4d3-b260d022fbf5" Jul 07 18:23:13.191987 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:13.191696 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k92pn" podUID="3f672f83-cb85-4c82-91fd-1d47b9c67d59" Jul 07 18:23:13.191987 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:13.191810 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-98qb4" podUID="e17ddfe4-73b3-45bd-b80f-ed992e7e1b71" Jul 07 18:23:13.292076 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:13.291952 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-66cjm" Jul 07 18:23:13.292636 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:13.292617 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-66cjm" Jul 07 18:23:14.766909 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:14.766855 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs\") pod \"network-metrics-daemon-bzlxp\" (UID: \"cde64ccc-edac-45b8-b4d3-b260d022fbf5\") " pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:14.767654 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:14.766918 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret\") pod \"global-pull-secret-syncer-98qb4\" (UID: \"e17ddfe4-73b3-45bd-b80f-ed992e7e1b71\") " pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:14.767654 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:14.767026 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 18:23:14.767654 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:14.767092 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret podName:e17ddfe4-73b3-45bd-b80f-ed992e7e1b71 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:30.7670719 +0000 UTC m=+34.150865299 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret") pod "global-pull-secret-syncer-98qb4" (UID: "e17ddfe4-73b3-45bd-b80f-ed992e7e1b71") : object "kube-system"/"original-pull-secret" not registered Jul 07 18:23:14.767654 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:14.767029 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:23:14.767654 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:14.767177 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs podName:cde64ccc-edac-45b8-b4d3-b260d022fbf5 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:30.767156083 +0000 UTC m=+34.150949485 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs") pod "network-metrics-daemon-bzlxp" (UID: "cde64ccc-edac-45b8-b4d3-b260d022fbf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:23:14.968991 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:14.968947 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kc7t\" (UniqueName: \"kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t\") pod \"network-check-target-k92pn\" (UID: \"3f672f83-cb85-4c82-91fd-1d47b9c67d59\") " pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:14.969148 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:14.969103 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 18:23:14.969148 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:14.969129 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 18:23:14.969148 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:14.969139 2578 projected.go:196] Error preparing data for projected volume kube-api-access-6kc7t for pod openshift-network-diagnostics/network-check-target-k92pn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:23:14.969282 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:14.969191 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t podName:3f672f83-cb85-4c82-91fd-1d47b9c67d59 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:30.969176415 +0000 UTC m=+34.352969792 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kc7t" (UniqueName: "kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t") pod "network-check-target-k92pn" (UID: "3f672f83-cb85-4c82-91fd-1d47b9c67d59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:23:15.191996 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:15.191904 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:15.192148 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:15.191917 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:15.192148 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:15.192038 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k92pn" podUID="3f672f83-cb85-4c82-91fd-1d47b9c67d59" Jul 07 18:23:15.192148 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:15.192130 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzlxp" podUID="cde64ccc-edac-45b8-b4d3-b260d022fbf5" Jul 07 18:23:15.192243 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:15.192177 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:15.192289 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:15.192242 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-98qb4" podUID="e17ddfe4-73b3-45bd-b80f-ed992e7e1b71" Jul 07 18:23:17.192423 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:17.192387 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:17.193083 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:17.192436 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:17.193083 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:17.192504 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:17.193083 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:17.192535 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-98qb4" podUID="e17ddfe4-73b3-45bd-b80f-ed992e7e1b71" Jul 07 18:23:17.193083 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:17.192592 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzlxp" podUID="cde64ccc-edac-45b8-b4d3-b260d022fbf5" Jul 07 18:23:17.193083 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:17.192679 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k92pn" podUID="3f672f83-cb85-4c82-91fd-1d47b9c67d59" Jul 07 18:23:19.191598 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:19.191562 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:19.192065 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:19.191562 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:19.192065 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:19.191675 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k92pn" podUID="3f672f83-cb85-4c82-91fd-1d47b9c67d59" Jul 07 18:23:19.192065 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:19.191562 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:19.192065 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:19.191771 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzlxp" podUID="cde64ccc-edac-45b8-b4d3-b260d022fbf5" Jul 07 18:23:19.192065 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:19.191860 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-98qb4" podUID="e17ddfe4-73b3-45bd-b80f-ed992e7e1b71" Jul 07 18:23:20.797991 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:20.797780 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 18:23:21.101800 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.101548 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T18:23:20.797990917Z","UUID":"82190d96-6cdd-416e-bac2-0d1e8e157414","Handler":null,"Name":"","Endpoint":""} Jul 07 18:23:21.103091 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.103060 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 18:23:21.103210 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.103113 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 18:23:21.191719 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.191693 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:21.191856 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.191693 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:21.191856 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:21.191824 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzlxp" podUID="cde64ccc-edac-45b8-b4d3-b260d022fbf5" Jul 07 18:23:21.191965 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.191870 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:21.192011 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:21.191966 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k92pn" podUID="3f672f83-cb85-4c82-91fd-1d47b9c67d59" Jul 07 18:23:21.192066 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:21.192050 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-98qb4" podUID="e17ddfe4-73b3-45bd-b80f-ed992e7e1b71" Jul 07 18:23:21.312941 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.312912 2578 generic.go:356] "Generic (PLEG): container finished" podID="f1e7507b-7367-4253-bb48-212f27b5f8ca" containerID="e80336cb06ae1dcd55aa8509acbf1112136cbdec82656ec42b5c721f9e4282b7" exitCode=0 Jul 07 18:23:21.313068 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.312994 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wgd4k" event={"ID":"f1e7507b-7367-4253-bb48-212f27b5f8ca","Type":"ContainerDied","Data":"e80336cb06ae1dcd55aa8509acbf1112136cbdec82656ec42b5c721f9e4282b7"} Jul 07 18:23:21.315782 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.315766 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 18:23:21.316127 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.316106 2578 generic.go:356] "Generic (PLEG): container finished" podID="1d688309-1ae2-40ac-9de5-3b37cf893695" containerID="4cba0901de62a1ef70ca653432c0bffa1866aa830a3032cded9f6fd83567744c" exitCode=1 Jul 07 18:23:21.316202 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.316181 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" event={"ID":"1d688309-1ae2-40ac-9de5-3b37cf893695","Type":"ContainerStarted","Data":"215885929c7c816503879f3d0a45c581b8fe08028c6f3576df8fb79173db9097"} Jul 07 18:23:21.316257 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.316237 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" event={"ID":"1d688309-1ae2-40ac-9de5-3b37cf893695","Type":"ContainerStarted","Data":"8b9c0e1ae11fe566d9fa2cb87930660a263a7eb365b4ea8b97eb94c5da03a164"} Jul 07 18:23:21.316308 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.316269 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" event={"ID":"1d688309-1ae2-40ac-9de5-3b37cf893695","Type":"ContainerStarted","Data":"8d6332108575e891981e69e52c6f6b3395f849512b9016c25b529da14cd1a0e9"} Jul 07 18:23:21.316381 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.316309 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" event={"ID":"1d688309-1ae2-40ac-9de5-3b37cf893695","Type":"ContainerDied","Data":"4cba0901de62a1ef70ca653432c0bffa1866aa830a3032cded9f6fd83567744c"} Jul 07 18:23:21.316381 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.316358 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" event={"ID":"1d688309-1ae2-40ac-9de5-3b37cf893695","Type":"ContainerStarted","Data":"ec8c8dacb1e7ca0124ede424b50077b52aed3adc5a7fb93ee05b1f1ba20cb100"} Jul 07 18:23:21.317584 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.317564 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9m4cn" event={"ID":"c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448","Type":"ContainerStarted","Data":"9944e7f1e88590064e745fd484c793cd96a7474257f798a9021153c0073dbe77"} Jul 07 18:23:21.319357 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.319339 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" event={"ID":"936682ff-fe11-4ce3-8d8c-98c7b909b439","Type":"ContainerStarted","Data":"ba1125938ac7deb712d1ce897f1cfed51868ef1b78e4668bcff3effe230479dc"} Jul 07 18:23:21.362805 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.362743 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9m4cn" podStartSLOduration=3.691353385 podStartE2EDuration="24.362721734s" totalImagesPullingTime="20.671368349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:21.36227253 +0000 UTC m=+24.746065927" watchObservedRunningTime="2026-07-07 18:23:21.362721734 +0000 UTC m=+24.746515130" Jul 07 18:23:21.544531 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.544458 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:21.715776 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.715736 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:21.842945 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:21.842855 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:22.008443 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:22.008408 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:22.325095 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:22.325062 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 18:23:22.325536 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:22.325509 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" event={"ID":"1d688309-1ae2-40ac-9de5-3b37cf893695","Type":"ContainerStarted","Data":"964fa9ede77ba11129087c041272c3e685dcbb5750c1de005fe5a6c9a3029816"} Jul 07 18:23:22.327955 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:22.327924 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" event={"ID":"936682ff-fe11-4ce3-8d8c-98c7b909b439","Type":"ContainerStarted","Data":"b48bf1f8c028cd3d540a7a347969085e492a582642e269eebee917227f166d7d"} Jul 07 18:23:23.191431 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:23.191403 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:23.191431 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:23.191427 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:23.192000 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:23.191497 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:23.192000 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:23.191510 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzlxp" podUID="cde64ccc-edac-45b8-b4d3-b260d022fbf5" Jul 07 18:23:23.192000 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:23.191566 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-98qb4" podUID="e17ddfe4-73b3-45bd-b80f-ed992e7e1b71" Jul 07 18:23:23.192000 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:23.191643 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k92pn" podUID="3f672f83-cb85-4c82-91fd-1d47b9c67d59" Jul 07 18:23:23.335958 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:23.335869 2578 generic.go:356] "Generic (PLEG): container finished" podID="f1e7507b-7367-4253-bb48-212f27b5f8ca" containerID="05362c31c1b17e08dc54027931871f0ec4a9d3d6fae8defd8c919163c60d3128" exitCode=0 Jul 07 18:23:23.335958 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:23.335944 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wgd4k" event={"ID":"f1e7507b-7367-4253-bb48-212f27b5f8ca","Type":"ContainerDied","Data":"05362c31c1b17e08dc54027931871f0ec4a9d3d6fae8defd8c919163c60d3128"} Jul 07 18:23:23.366930 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:23.366868 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8vn8" podStartSLOduration=6.20095911 podStartE2EDuration="26.366852224s" totalImagesPullingTime="20.165893114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:22.36544629 +0000 UTC m=+25.749239737" watchObservedRunningTime="2026-07-07 18:23:23.366852224 +0000 UTC m=+26.750645619" Jul 07 18:23:23.649574 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:23.649495 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:24.340102 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:24.340017 2578 generic.go:356] "Generic (PLEG): container finished" podID="f1e7507b-7367-4253-bb48-212f27b5f8ca" containerID="6538a5d8f822c1dfdcdee490c24154dbdb03516a1db3afc559bcfdcc4614d211" exitCode=0 Jul 07 18:23:24.340534 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:24.340106 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wgd4k" event={"ID":"f1e7507b-7367-4253-bb48-212f27b5f8ca","Type":"ContainerDied","Data":"6538a5d8f822c1dfdcdee490c24154dbdb03516a1db3afc559bcfdcc4614d211"} Jul 07 18:23:24.343211 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:24.343193 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 18:23:24.343640 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:24.343620 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" event={"ID":"1d688309-1ae2-40ac-9de5-3b37cf893695","Type":"ContainerStarted","Data":"1e8aa27e08fce490307458aa60db920d361ed29803046472845b9eb8e713cfbd"} Jul 07 18:23:24.541569 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:24.541528 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:25.191898 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:25.191868 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:25.192100 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:25.191869 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:25.192100 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:25.192002 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k92pn" podUID="3f672f83-cb85-4c82-91fd-1d47b9c67d59" Jul 07 18:23:25.192100 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:25.191870 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:25.192265 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:25.192098 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzlxp" podUID="cde64ccc-edac-45b8-b4d3-b260d022fbf5" Jul 07 18:23:25.192265 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:25.192152 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-98qb4" podUID="e17ddfe4-73b3-45bd-b80f-ed992e7e1b71" Jul 07 18:23:26.354529 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:26.354244 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 18:23:26.355483 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:26.355445 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" event={"ID":"1d688309-1ae2-40ac-9de5-3b37cf893695","Type":"ContainerStarted","Data":"3c2447619ce37d80b5249b2ba56e4c4c4d389965618efd351317488e2ab46c61"} Jul 07 18:23:26.355777 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:26.355754 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:23:26.355894 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:26.355832 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:23:26.356008 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:26.355991 2578 scope.go:117] "RemoveContainer" containerID="4cba0901de62a1ef70ca653432c0bffa1866aa830a3032cded9f6fd83567744c" Jul 07 18:23:26.376199 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:26.376177 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:23:27.195299 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:27.195271 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:27.195499 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:27.195383 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:27.195499 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:27.195389 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzlxp" podUID="cde64ccc-edac-45b8-b4d3-b260d022fbf5" Jul 07 18:23:27.195499 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:27.195411 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:27.195650 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:27.195510 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-98qb4" podUID="e17ddfe4-73b3-45bd-b80f-ed992e7e1b71" Jul 07 18:23:27.195650 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:27.195594 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k92pn" podUID="3f672f83-cb85-4c82-91fd-1d47b9c67d59" Jul 07 18:23:27.363507 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:27.363478 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 18:23:27.364135 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:27.364104 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" event={"ID":"1d688309-1ae2-40ac-9de5-3b37cf893695","Type":"ContainerStarted","Data":"6e16ac9859ce149425cb7fe598247a10baddd4d69f55a597712ed0b6620ac570"} Jul 07 18:23:27.364703 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:27.364607 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:23:27.383089 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:27.383057 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:23:27.401592 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:27.401528 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" podStartSLOduration=11.303468254 podStartE2EDuration="30.401505837s" totalImagesPullingTime="19.098037583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:27.398287618 +0000 UTC m=+30.782081051" watchObservedRunningTime="2026-07-07 18:23:27.401505837 +0000 UTC m=+30.785299231" Jul 07 18:23:27.704915 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:27.704851 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:27.839200 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:27.839150 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:27.855905 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:27.855882 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k92pn"] Jul 07 18:23:27.856029 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:27.856010 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:27.856105 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:27.856089 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k92pn" podUID="3f672f83-cb85-4c82-91fd-1d47b9c67d59" Jul 07 18:23:27.859135 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:27.859081 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bzlxp"] Jul 07 18:23:27.859279 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:27.859246 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:27.859440 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:27.859383 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzlxp" podUID="cde64ccc-edac-45b8-b4d3-b260d022fbf5" Jul 07 18:23:27.859991 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:27.859969 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-98qb4"] Jul 07 18:23:27.860111 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:27.860072 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:27.860191 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:27.860167 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-98qb4" podUID="e17ddfe4-73b3-45bd-b80f-ed992e7e1b71" Jul 07 18:23:29.194932 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:29.194900 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:29.195349 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:29.194939 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:29.195349 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:29.194939 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:29.195349 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:29.195005 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzlxp" podUID="cde64ccc-edac-45b8-b4d3-b260d022fbf5" Jul 07 18:23:29.195349 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:29.195098 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k92pn" podUID="3f672f83-cb85-4c82-91fd-1d47b9c67d59" Jul 07 18:23:29.195349 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:29.195184 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-98qb4" podUID="e17ddfe4-73b3-45bd-b80f-ed992e7e1b71" Jul 07 18:23:29.702463 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:29.702428 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:29.791815 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:29.791775 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:30.030870 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:30.030840 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:30.099128 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:30.099085 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:30.395534 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:30.395436 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:30.482383 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:30.482346 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:30.717989 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:30.717905 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:30.791099 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:30.791066 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs\") pod \"network-metrics-daemon-bzlxp\" (UID: \"cde64ccc-edac-45b8-b4d3-b260d022fbf5\") " pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:30.791099 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:30.791108 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret\") pod \"global-pull-secret-syncer-98qb4\" (UID: \"e17ddfe4-73b3-45bd-b80f-ed992e7e1b71\") " pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:30.791355 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:30.791238 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 18:23:30.791355 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:30.791242 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:23:30.791355 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:30.791305 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret podName:e17ddfe4-73b3-45bd-b80f-ed992e7e1b71 nodeName:}" failed. No retries permitted until 2026-07-07 18:24:02.791285983 +0000 UTC m=+66.175079356 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret") pod "global-pull-secret-syncer-98qb4" (UID: "e17ddfe4-73b3-45bd-b80f-ed992e7e1b71") : object "kube-system"/"original-pull-secret" not registered Jul 07 18:23:30.791355 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:30.791338 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs podName:cde64ccc-edac-45b8-b4d3-b260d022fbf5 nodeName:}" failed. No retries permitted until 2026-07-07 18:24:02.791314517 +0000 UTC m=+66.175107889 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs") pod "network-metrics-daemon-bzlxp" (UID: "cde64ccc-edac-45b8-b4d3-b260d022fbf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 18:23:30.816047 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:30.815996 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:30.920158 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:30.920123 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:30.992337 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:30.992291 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kc7t\" (UniqueName: \"kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t\") pod \"network-check-target-k92pn\" (UID: \"3f672f83-cb85-4c82-91fd-1d47b9c67d59\") " pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:30.992498 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:30.992474 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 18:23:30.992553 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:30.992501 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 18:23:30.992553 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:30.992515 2578 projected.go:196] Error preparing data for projected volume kube-api-access-6kc7t for pod openshift-network-diagnostics/network-check-target-k92pn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:23:30.992612 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:30.992573 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t podName:3f672f83-cb85-4c82-91fd-1d47b9c67d59 nodeName:}" failed. No retries permitted until 2026-07-07 18:24:02.992555896 +0000 UTC m=+66.376349279 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kc7t" (UniqueName: "kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t") pod "network-check-target-k92pn" (UID: "3f672f83-cb85-4c82-91fd-1d47b9c67d59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 18:23:31.141604 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:31.141565 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:31.196157 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:31.196125 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:31.196157 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:31.196141 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:31.196383 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:31.196237 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:31.196383 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:31.196233 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k92pn" podUID="3f672f83-cb85-4c82-91fd-1d47b9c67d59" Jul 07 18:23:31.196485 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:31.196396 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-98qb4" podUID="e17ddfe4-73b3-45bd-b80f-ed992e7e1b71" Jul 07 18:23:31.196485 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:31.196418 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzlxp" podUID="cde64ccc-edac-45b8-b4d3-b260d022fbf5" Jul 07 18:23:31.250700 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:31.250581 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:31.359161 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:31.359136 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bnkff_bf3379f6-bbce-4398-bb71-1dbe3ddc5059/dns-node-resolver/0.log" Jul 07 18:23:32.378904 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:32.378679 2578 generic.go:356] "Generic (PLEG): container finished" podID="f1e7507b-7367-4253-bb48-212f27b5f8ca" containerID="7cb9d2d4b46e8d0fff650d221e77ff893d8b145259be0ad1bede4a23a6933441" exitCode=0 Jul 07 18:23:32.379383 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:32.378760 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wgd4k" event={"ID":"f1e7507b-7367-4253-bb48-212f27b5f8ca","Type":"ContainerDied","Data":"7cb9d2d4b46e8d0fff650d221e77ff893d8b145259be0ad1bede4a23a6933441"} Jul 07 18:23:32.504078 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:32.504050 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:32.624306 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:32.624270 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:32.720513 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:32.720488 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:32.752294 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:32.752267 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wlm6k_d3f4cdc4-da50-4c22-8e51-55872deaf1c8/node-ca/0.log" Jul 07 18:23:32.797712 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:32.797681 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:33.194216 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:33.194187 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:33.194414 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:33.194274 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:33.194414 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:33.194282 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k92pn" podUID="3f672f83-cb85-4c82-91fd-1d47b9c67d59" Jul 07 18:23:33.194414 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:33.194383 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzlxp" podUID="cde64ccc-edac-45b8-b4d3-b260d022fbf5" Jul 07 18:23:33.194517 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:33.194424 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:33.194517 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:23:33.194476 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-98qb4" podUID="e17ddfe4-73b3-45bd-b80f-ed992e7e1b71" Jul 07 18:23:33.348402 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:33.348311 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-108.ec2.internal" event="NodeReady" Jul 07 18:23:33.348547 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:33.348444 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 18:23:33.387254 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:33.387221 2578 generic.go:356] "Generic (PLEG): container finished" podID="f1e7507b-7367-4253-bb48-212f27b5f8ca" containerID="829586a7e135a7cc3618cf452732ca3fcec6dc467a6f4b7cdb3ac2ec51c6bdf1" exitCode=0 Jul 07 18:23:33.387254 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:33.387259 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wgd4k" event={"ID":"f1e7507b-7367-4253-bb48-212f27b5f8ca","Type":"ContainerDied","Data":"829586a7e135a7cc3618cf452732ca3fcec6dc467a6f4b7cdb3ac2ec51c6bdf1"} Jul 07 18:23:33.539817 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:33.539788 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:34.393227 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:34.393192 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wgd4k" event={"ID":"f1e7507b-7367-4253-bb48-212f27b5f8ca","Type":"ContainerStarted","Data":"c20bb217f5d395d6d36e68958dff9c2ecc99b4083f8bafe168deb98a8860e1d3"} Jul 07 18:23:34.436766 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:34.436713 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wgd4k" podStartSLOduration=7.717656401 podStartE2EDuration="37.436697917s" totalImagesPullingTime="29.719041516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:34.435579952 +0000 UTC m=+37.819373346" watchObservedRunningTime="2026-07-07 18:23:34.436697917 +0000 UTC m=+37.820491310" Jul 07 18:23:35.195019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:35.194989 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:23:35.195212 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:35.195090 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:23:35.195212 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:35.195115 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:23:35.198749 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:35.198726 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 18:23:35.199480 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:35.199460 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dxsjk\"" Jul 07 18:23:35.199615 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:35.199534 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 18:23:35.199615 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:35.199586 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 18:23:35.199814 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:35.199798 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 18:23:35.199880 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:35.199820 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wxf5l\"" Jul 07 18:23:36.580531 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:36.580487 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:36.685239 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:36.685207 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:36.799531 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:36.799496 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:37.899594 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:37.899552 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:38.042691 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:38.042662 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:38.187050 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:38.186971 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:38.372187 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:38.372156 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:38.535110 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:38.535082 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:38.868528 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:38.868447 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:39.041658 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:39.041629 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:39.409806 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:39.409771 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:39.489224 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:39.489196 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:40.224637 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:40.224604 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:40.459081 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:40.459046 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:40.663416 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:40.663312 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:40.820998 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:40.820968 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:41.031825 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:41.031790 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:41.555813 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:41.555767 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:41.656418 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:41.656382 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:41.830550 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:41.830457 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:42.445196 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:42.445166 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:42.541473 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:42.541434 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:42.645107 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:42.645064 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:42.798301 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:42.798263 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:43.030724 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:43.030690 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:43.236667 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:43.236637 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:43.534537 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:43.534446 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:43.772180 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:43.772147 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:44.224645 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:44.224610 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:44.330526 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:44.330495 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:44.830625 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:44.830586 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:45.037478 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:45.037444 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:45.328795 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:45.328765 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:45.430104 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:45.430071 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:45.632981 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:45.632903 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:46.243544 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:46.243508 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:46.451866 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:46.451833 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:47.031620 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:47.031588 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:47.278064 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:47.278032 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:47.629492 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:47.629458 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:48.232457 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:48.232424 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:48.444060 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:48.444027 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:48.639151 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:48.639065 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:51.567084 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:51.567051 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:51.641312 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:51.641286 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:56.086703 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.086663 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pd2qx"] Jul 07 18:23:56.089155 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.089139 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pd2qx" Jul 07 18:23:56.093620 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.093585 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 18:23:56.093793 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.093772 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 18:23:56.096054 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.096033 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 18:23:56.096243 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.096200 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 18:23:56.096558 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.096542 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-62q56\"" Jul 07 18:23:56.113808 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.113779 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pd2qx"] Jul 07 18:23:56.158334 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.158299 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d4np\" (UniqueName: \"kubernetes.io/projected/5e5a23b5-a6ca-4693-afc4-489102504294-kube-api-access-7d4np\") pod \"insights-runtime-extractor-pd2qx\" (UID: \"5e5a23b5-a6ca-4693-afc4-489102504294\") " pod="openshift-insights/insights-runtime-extractor-pd2qx" Jul 07 18:23:56.158507 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.158345 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e5a23b5-a6ca-4693-afc4-489102504294-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd2qx\" (UID: \"5e5a23b5-a6ca-4693-afc4-489102504294\") " pod="openshift-insights/insights-runtime-extractor-pd2qx" Jul 07 18:23:56.158507 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.158432 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e5a23b5-a6ca-4693-afc4-489102504294-data-volume\") pod \"insights-runtime-extractor-pd2qx\" (UID: \"5e5a23b5-a6ca-4693-afc4-489102504294\") " pod="openshift-insights/insights-runtime-extractor-pd2qx" Jul 07 18:23:56.158507 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.158474 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e5a23b5-a6ca-4693-afc4-489102504294-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd2qx\" (UID: \"5e5a23b5-a6ca-4693-afc4-489102504294\") " pod="openshift-insights/insights-runtime-extractor-pd2qx" Jul 07 18:23:56.158616 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.158511 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e5a23b5-a6ca-4693-afc4-489102504294-crio-socket\") pod \"insights-runtime-extractor-pd2qx\" (UID: \"5e5a23b5-a6ca-4693-afc4-489102504294\") " pod="openshift-insights/insights-runtime-extractor-pd2qx" Jul 07 18:23:56.179335 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.179295 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sxzvf"] Jul 07 18:23:56.181283 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.181257 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sxzvf" Jul 07 18:23:56.185242 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.185219 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 18:23:56.185730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.185714 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xk8xw\"" Jul 07 18:23:56.187177 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.187152 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 18:23:56.188147 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.188064 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 18:23:56.188410 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.188393 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth"] Jul 07 18:23:56.190330 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.190305 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" Jul 07 18:23:56.193910 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.193890 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b68d7f5b-5hs6w"] Jul 07 18:23:56.195995 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.195978 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b68d7f5b-5hs6w" Jul 07 18:23:56.196480 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.196459 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 07 18:23:56.196799 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.196779 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 07 18:23:56.197766 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.197750 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 07 18:23:56.197862 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.197780 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 07 18:23:56.198030 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.198016 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-rz2wq\"" Jul 07 18:23:56.205575 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.205558 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 07 18:23:56.205657 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.205606 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 07 18:23:56.205657 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.205566 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 07 18:23:56.205657 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.205568 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 07 18:23:56.208539 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.208520 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fdf557895-96fn2"] Jul 07 18:23:56.210451 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.210430 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fdf557895-96fn2" Jul 07 18:23:56.213225 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.213203 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sxzvf"] Jul 07 18:23:56.213451 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.213438 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 07 18:23:56.219543 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.219524 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth"] Jul 07 18:23:56.231333 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.231299 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b68d7f5b-5hs6w"] Jul 07 18:23:56.242538 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.242514 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fdf557895-96fn2"] Jul 07 18:23:56.259580 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.259546 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b1cd385b-fabe-4053-bdcb-9fc07293d8c8-hub\") pod \"cluster-proxy-proxy-agent-85995f898d-wtbth\" (UID: \"b1cd385b-fabe-4053-bdcb-9fc07293d8c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" Jul 07 18:23:56.259580 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.259583 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1cd385b-fabe-4053-bdcb-9fc07293d8c8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85995f898d-wtbth\" (UID: \"b1cd385b-fabe-4053-bdcb-9fc07293d8c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" Jul 07 18:23:56.259798 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.259604 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65wc7\" (UniqueName: \"kubernetes.io/projected/b1cd385b-fabe-4053-bdcb-9fc07293d8c8-kube-api-access-65wc7\") pod \"cluster-proxy-proxy-agent-85995f898d-wtbth\" (UID: \"b1cd385b-fabe-4053-bdcb-9fc07293d8c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" Jul 07 18:23:56.259798 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.259624 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e5a23b5-a6ca-4693-afc4-489102504294-data-volume\") pod \"insights-runtime-extractor-pd2qx\" (UID: \"5e5a23b5-a6ca-4693-afc4-489102504294\") " pod="openshift-insights/insights-runtime-extractor-pd2qx" Jul 07 18:23:56.259798 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.259640 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e5a23b5-a6ca-4693-afc4-489102504294-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd2qx\" (UID: \"5e5a23b5-a6ca-4693-afc4-489102504294\") " pod="openshift-insights/insights-runtime-extractor-pd2qx" Jul 07 18:23:56.259798 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.259663 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgpkv\" (UniqueName: \"kubernetes.io/projected/e2838a03-723b-46e2-8a9f-db1a1625220c-kube-api-access-rgpkv\") pod \"klusterlet-addon-workmgr-5fdf557895-96fn2\" (UID: \"e2838a03-723b-46e2-8a9f-db1a1625220c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fdf557895-96fn2" Jul 07 18:23:56.259798 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.259682 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b1cd385b-fabe-4053-bdcb-9fc07293d8c8-ca\") pod \"cluster-proxy-proxy-agent-85995f898d-wtbth\" (UID: \"b1cd385b-fabe-4053-bdcb-9fc07293d8c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" Jul 07 18:23:56.259798 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.259696 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b1cd385b-fabe-4053-bdcb-9fc07293d8c8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85995f898d-wtbth\" (UID: \"b1cd385b-fabe-4053-bdcb-9fc07293d8c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" Jul 07 18:23:56.259798 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.259711 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be81195f-a6f0-4154-abde-ccaa50f68346-cert\") pod \"ingress-canary-sxzvf\" (UID: \"be81195f-a6f0-4154-abde-ccaa50f68346\") " pod="openshift-ingress-canary/ingress-canary-sxzvf" Jul 07 18:23:56.259798 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.259733 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6scl\" (UniqueName: \"kubernetes.io/projected/4588b8b8-62be-4d7e-b4a3-8686b47a9d21-kube-api-access-z6scl\") pod \"managed-serviceaccount-addon-agent-6b68d7f5b-5hs6w\" (UID: \"4588b8b8-62be-4d7e-b4a3-8686b47a9d21\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b68d7f5b-5hs6w" Jul 07 18:23:56.260076 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.259843 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e5a23b5-a6ca-4693-afc4-489102504294-crio-socket\") pod \"insights-runtime-extractor-pd2qx\" (UID: \"5e5a23b5-a6ca-4693-afc4-489102504294\") " pod="openshift-insights/insights-runtime-extractor-pd2qx" Jul 07 18:23:56.260076 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.259873 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bxxm\" (UniqueName: \"kubernetes.io/projected/be81195f-a6f0-4154-abde-ccaa50f68346-kube-api-access-7bxxm\") pod \"ingress-canary-sxzvf\" (UID: \"be81195f-a6f0-4154-abde-ccaa50f68346\") " pod="openshift-ingress-canary/ingress-canary-sxzvf" Jul 07 18:23:56.260076 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.259895 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2838a03-723b-46e2-8a9f-db1a1625220c-tmp\") pod \"klusterlet-addon-workmgr-5fdf557895-96fn2\" (UID: \"e2838a03-723b-46e2-8a9f-db1a1625220c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fdf557895-96fn2" Jul 07 18:23:56.260076 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.259915 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4588b8b8-62be-4d7e-b4a3-8686b47a9d21-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b68d7f5b-5hs6w\" (UID: \"4588b8b8-62be-4d7e-b4a3-8686b47a9d21\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b68d7f5b-5hs6w" Jul 07 18:23:56.260076 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.259921 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e5a23b5-a6ca-4693-afc4-489102504294-data-volume\") pod \"insights-runtime-extractor-pd2qx\" (UID: \"5e5a23b5-a6ca-4693-afc4-489102504294\") " pod="openshift-insights/insights-runtime-extractor-pd2qx" Jul 07 18:23:56.260076 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.259978 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d4np\" (UniqueName: \"kubernetes.io/projected/5e5a23b5-a6ca-4693-afc4-489102504294-kube-api-access-7d4np\") pod \"insights-runtime-extractor-pd2qx\" (UID: \"5e5a23b5-a6ca-4693-afc4-489102504294\") " pod="openshift-insights/insights-runtime-extractor-pd2qx" Jul 07 18:23:56.260076 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.260035 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e5a23b5-a6ca-4693-afc4-489102504294-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd2qx\" (UID: \"5e5a23b5-a6ca-4693-afc4-489102504294\") " pod="openshift-insights/insights-runtime-extractor-pd2qx" Jul 07 18:23:56.260076 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.260059 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e5a23b5-a6ca-4693-afc4-489102504294-crio-socket\") pod \"insights-runtime-extractor-pd2qx\" (UID: \"5e5a23b5-a6ca-4693-afc4-489102504294\") " pod="openshift-insights/insights-runtime-extractor-pd2qx" Jul 07 18:23:56.260356 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.260131 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1cd385b-fabe-4053-bdcb-9fc07293d8c8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85995f898d-wtbth\" (UID: \"b1cd385b-fabe-4053-bdcb-9fc07293d8c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" Jul 07 18:23:56.260356 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.260195 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e2838a03-723b-46e2-8a9f-db1a1625220c-klusterlet-config\") pod \"klusterlet-addon-workmgr-5fdf557895-96fn2\" (UID: \"e2838a03-723b-46e2-8a9f-db1a1625220c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fdf557895-96fn2" Jul 07 18:23:56.260356 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.260204 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e5a23b5-a6ca-4693-afc4-489102504294-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd2qx\" (UID: \"5e5a23b5-a6ca-4693-afc4-489102504294\") " pod="openshift-insights/insights-runtime-extractor-pd2qx" Jul 07 18:23:56.263925 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.263907 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e5a23b5-a6ca-4693-afc4-489102504294-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd2qx\" (UID: \"5e5a23b5-a6ca-4693-afc4-489102504294\") " pod="openshift-insights/insights-runtime-extractor-pd2qx" Jul 07 18:23:56.289590 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.289562 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d4np\" (UniqueName: \"kubernetes.io/projected/5e5a23b5-a6ca-4693-afc4-489102504294-kube-api-access-7d4np\") pod \"insights-runtime-extractor-pd2qx\" (UID: \"5e5a23b5-a6ca-4693-afc4-489102504294\") " pod="openshift-insights/insights-runtime-extractor-pd2qx" Jul 07 18:23:56.294270 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.294239 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bncsv"] Jul 07 18:23:56.296352 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.296316 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bncsv" Jul 07 18:23:56.300164 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.300147 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 18:23:56.303525 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.303507 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9fz64\"" Jul 07 18:23:56.304479 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.304463 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 18:23:56.319215 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.319190 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bncsv"] Jul 07 18:23:56.320750 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.320727 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:56.360968 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.360878 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bxxm\" (UniqueName: \"kubernetes.io/projected/be81195f-a6f0-4154-abde-ccaa50f68346-kube-api-access-7bxxm\") pod \"ingress-canary-sxzvf\" (UID: \"be81195f-a6f0-4154-abde-ccaa50f68346\") " pod="openshift-ingress-canary/ingress-canary-sxzvf" Jul 07 18:23:56.360968 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.360915 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2838a03-723b-46e2-8a9f-db1a1625220c-tmp\") pod \"klusterlet-addon-workmgr-5fdf557895-96fn2\" (UID: \"e2838a03-723b-46e2-8a9f-db1a1625220c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fdf557895-96fn2" Jul 07 18:23:56.360968 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.360935 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4588b8b8-62be-4d7e-b4a3-8686b47a9d21-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b68d7f5b-5hs6w\" (UID: \"4588b8b8-62be-4d7e-b4a3-8686b47a9d21\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b68d7f5b-5hs6w" Jul 07 18:23:56.361235 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.360973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1cd385b-fabe-4053-bdcb-9fc07293d8c8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85995f898d-wtbth\" (UID: \"b1cd385b-fabe-4053-bdcb-9fc07293d8c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" Jul 07 18:23:56.361235 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.360995 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e2838a03-723b-46e2-8a9f-db1a1625220c-klusterlet-config\") pod \"klusterlet-addon-workmgr-5fdf557895-96fn2\" (UID: \"e2838a03-723b-46e2-8a9f-db1a1625220c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fdf557895-96fn2" Jul 07 18:23:56.361235 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.361026 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-652s5\" (UniqueName: \"kubernetes.io/projected/03509627-aac5-42cf-a76b-4b7e499bdb4e-kube-api-access-652s5\") pod \"dns-default-bncsv\" (UID: \"03509627-aac5-42cf-a76b-4b7e499bdb4e\") " pod="openshift-dns/dns-default-bncsv" Jul 07 18:23:56.361235 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.361061 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b1cd385b-fabe-4053-bdcb-9fc07293d8c8-hub\") pod \"cluster-proxy-proxy-agent-85995f898d-wtbth\" (UID: \"b1cd385b-fabe-4053-bdcb-9fc07293d8c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" Jul 07 18:23:56.361235 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.361083 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1cd385b-fabe-4053-bdcb-9fc07293d8c8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85995f898d-wtbth\" (UID: \"b1cd385b-fabe-4053-bdcb-9fc07293d8c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" Jul 07 18:23:56.361235 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.361105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65wc7\" (UniqueName: \"kubernetes.io/projected/b1cd385b-fabe-4053-bdcb-9fc07293d8c8-kube-api-access-65wc7\") pod \"cluster-proxy-proxy-agent-85995f898d-wtbth\" (UID: \"b1cd385b-fabe-4053-bdcb-9fc07293d8c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" Jul 07 18:23:56.361235 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.361132 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgpkv\" (UniqueName: \"kubernetes.io/projected/e2838a03-723b-46e2-8a9f-db1a1625220c-kube-api-access-rgpkv\") pod \"klusterlet-addon-workmgr-5fdf557895-96fn2\" (UID: \"e2838a03-723b-46e2-8a9f-db1a1625220c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fdf557895-96fn2" Jul 07 18:23:56.361235 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.361147 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03509627-aac5-42cf-a76b-4b7e499bdb4e-metrics-tls\") pod \"dns-default-bncsv\" (UID: \"03509627-aac5-42cf-a76b-4b7e499bdb4e\") " pod="openshift-dns/dns-default-bncsv" Jul 07 18:23:56.361235 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.361161 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03509627-aac5-42cf-a76b-4b7e499bdb4e-tmp-dir\") pod \"dns-default-bncsv\" (UID: \"03509627-aac5-42cf-a76b-4b7e499bdb4e\") " pod="openshift-dns/dns-default-bncsv" Jul 07 18:23:56.361235 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.361178 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b1cd385b-fabe-4053-bdcb-9fc07293d8c8-ca\") pod \"cluster-proxy-proxy-agent-85995f898d-wtbth\" (UID: \"b1cd385b-fabe-4053-bdcb-9fc07293d8c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" Jul 07 18:23:56.361235 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.361199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b1cd385b-fabe-4053-bdcb-9fc07293d8c8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85995f898d-wtbth\" (UID: \"b1cd385b-fabe-4053-bdcb-9fc07293d8c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" Jul 07 18:23:56.361235 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.361217 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be81195f-a6f0-4154-abde-ccaa50f68346-cert\") pod \"ingress-canary-sxzvf\" (UID: \"be81195f-a6f0-4154-abde-ccaa50f68346\") " pod="openshift-ingress-canary/ingress-canary-sxzvf" Jul 07 18:23:56.361800 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.361245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6scl\" (UniqueName: \"kubernetes.io/projected/4588b8b8-62be-4d7e-b4a3-8686b47a9d21-kube-api-access-z6scl\") pod \"managed-serviceaccount-addon-agent-6b68d7f5b-5hs6w\" (UID: \"4588b8b8-62be-4d7e-b4a3-8686b47a9d21\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b68d7f5b-5hs6w" Jul 07 18:23:56.361800 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.361264 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/03509627-aac5-42cf-a76b-4b7e499bdb4e-config-volume\") pod \"dns-default-bncsv\" (UID: \"03509627-aac5-42cf-a76b-4b7e499bdb4e\") " pod="openshift-dns/dns-default-bncsv" Jul 07 18:23:56.361800 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.361445 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2838a03-723b-46e2-8a9f-db1a1625220c-tmp\") pod \"klusterlet-addon-workmgr-5fdf557895-96fn2\" (UID: \"e2838a03-723b-46e2-8a9f-db1a1625220c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fdf557895-96fn2" Jul 07 18:23:56.362183 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.362155 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1cd385b-fabe-4053-bdcb-9fc07293d8c8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85995f898d-wtbth\" (UID: \"b1cd385b-fabe-4053-bdcb-9fc07293d8c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" Jul 07 18:23:56.364076 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.364045 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e2838a03-723b-46e2-8a9f-db1a1625220c-klusterlet-config\") pod \"klusterlet-addon-workmgr-5fdf557895-96fn2\" (UID: \"e2838a03-723b-46e2-8a9f-db1a1625220c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fdf557895-96fn2" Jul 07 18:23:56.365072 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.365011 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4588b8b8-62be-4d7e-b4a3-8686b47a9d21-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b68d7f5b-5hs6w\" (UID: \"4588b8b8-62be-4d7e-b4a3-8686b47a9d21\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b68d7f5b-5hs6w" Jul 07 18:23:56.365175 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.365140 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be81195f-a6f0-4154-abde-ccaa50f68346-cert\") pod \"ingress-canary-sxzvf\" (UID: \"be81195f-a6f0-4154-abde-ccaa50f68346\") " pod="openshift-ingress-canary/ingress-canary-sxzvf" Jul 07 18:23:56.365760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.365737 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b1cd385b-fabe-4053-bdcb-9fc07293d8c8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85995f898d-wtbth\" (UID: \"b1cd385b-fabe-4053-bdcb-9fc07293d8c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" Jul 07 18:23:56.365907 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.365889 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1cd385b-fabe-4053-bdcb-9fc07293d8c8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85995f898d-wtbth\" (UID: \"b1cd385b-fabe-4053-bdcb-9fc07293d8c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" Jul 07 18:23:56.366254 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.366233 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b1cd385b-fabe-4053-bdcb-9fc07293d8c8-hub\") pod \"cluster-proxy-proxy-agent-85995f898d-wtbth\" (UID: \"b1cd385b-fabe-4053-bdcb-9fc07293d8c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" Jul 07 18:23:56.366401 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.366387 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b1cd385b-fabe-4053-bdcb-9fc07293d8c8-ca\") pod \"cluster-proxy-proxy-agent-85995f898d-wtbth\" (UID: \"b1cd385b-fabe-4053-bdcb-9fc07293d8c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" Jul 07 18:23:56.388722 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.388694 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bxxm\" (UniqueName: \"kubernetes.io/projected/be81195f-a6f0-4154-abde-ccaa50f68346-kube-api-access-7bxxm\") pod \"ingress-canary-sxzvf\" (UID: \"be81195f-a6f0-4154-abde-ccaa50f68346\") " pod="openshift-ingress-canary/ingress-canary-sxzvf" Jul 07 18:23:56.389921 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.389897 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgpkv\" (UniqueName: \"kubernetes.io/projected/e2838a03-723b-46e2-8a9f-db1a1625220c-kube-api-access-rgpkv\") pod \"klusterlet-addon-workmgr-5fdf557895-96fn2\" (UID: \"e2838a03-723b-46e2-8a9f-db1a1625220c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fdf557895-96fn2" Jul 07 18:23:56.390876 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.390854 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65wc7\" (UniqueName: \"kubernetes.io/projected/b1cd385b-fabe-4053-bdcb-9fc07293d8c8-kube-api-access-65wc7\") pod \"cluster-proxy-proxy-agent-85995f898d-wtbth\" (UID: \"b1cd385b-fabe-4053-bdcb-9fc07293d8c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" Jul 07 18:23:56.397122 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.397078 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6scl\" (UniqueName: \"kubernetes.io/projected/4588b8b8-62be-4d7e-b4a3-8686b47a9d21-kube-api-access-z6scl\") pod \"managed-serviceaccount-addon-agent-6b68d7f5b-5hs6w\" (UID: \"4588b8b8-62be-4d7e-b4a3-8686b47a9d21\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b68d7f5b-5hs6w" Jul 07 18:23:56.399191 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.399171 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pd2qx" Jul 07 18:23:56.462518 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.462486 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-652s5\" (UniqueName: \"kubernetes.io/projected/03509627-aac5-42cf-a76b-4b7e499bdb4e-kube-api-access-652s5\") pod \"dns-default-bncsv\" (UID: \"03509627-aac5-42cf-a76b-4b7e499bdb4e\") " pod="openshift-dns/dns-default-bncsv" Jul 07 18:23:56.462691 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.462550 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03509627-aac5-42cf-a76b-4b7e499bdb4e-metrics-tls\") pod \"dns-default-bncsv\" (UID: \"03509627-aac5-42cf-a76b-4b7e499bdb4e\") " pod="openshift-dns/dns-default-bncsv" Jul 07 18:23:56.462691 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.462577 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03509627-aac5-42cf-a76b-4b7e499bdb4e-tmp-dir\") pod \"dns-default-bncsv\" (UID: \"03509627-aac5-42cf-a76b-4b7e499bdb4e\") " pod="openshift-dns/dns-default-bncsv" Jul 07 18:23:56.462691 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.462604 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/03509627-aac5-42cf-a76b-4b7e499bdb4e-config-volume\") pod \"dns-default-bncsv\" (UID: \"03509627-aac5-42cf-a76b-4b7e499bdb4e\") " pod="openshift-dns/dns-default-bncsv" Jul 07 18:23:56.463017 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.462994 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03509627-aac5-42cf-a76b-4b7e499bdb4e-tmp-dir\") pod \"dns-default-bncsv\" (UID: \"03509627-aac5-42cf-a76b-4b7e499bdb4e\") " pod="openshift-dns/dns-default-bncsv" Jul 07 18:23:56.463309 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.463278 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/03509627-aac5-42cf-a76b-4b7e499bdb4e-config-volume\") pod \"dns-default-bncsv\" (UID: \"03509627-aac5-42cf-a76b-4b7e499bdb4e\") " pod="openshift-dns/dns-default-bncsv" Jul 07 18:23:56.464891 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.464868 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03509627-aac5-42cf-a76b-4b7e499bdb4e-metrics-tls\") pod \"dns-default-bncsv\" (UID: \"03509627-aac5-42cf-a76b-4b7e499bdb4e\") " pod="openshift-dns/dns-default-bncsv" Jul 07 18:23:56.489155 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.489120 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-652s5\" (UniqueName: \"kubernetes.io/projected/03509627-aac5-42cf-a76b-4b7e499bdb4e-kube-api-access-652s5\") pod \"dns-default-bncsv\" (UID: \"03509627-aac5-42cf-a76b-4b7e499bdb4e\") " pod="openshift-dns/dns-default-bncsv" Jul 07 18:23:56.489883 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.489869 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sxzvf" Jul 07 18:23:56.511261 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.511221 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" Jul 07 18:23:56.519778 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.519749 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b68d7f5b-5hs6w" Jul 07 18:23:56.536963 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.536918 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fdf557895-96fn2" Jul 07 18:23:56.571888 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.571832 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pd2qx"] Jul 07 18:23:56.602603 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.602530 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:56.605340 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.605255 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bncsv" Jul 07 18:23:56.773627 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.773601 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sxzvf"] Jul 07 18:23:56.774965 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:23:56.774937 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe81195f_a6f0_4154_abde_ccaa50f68346.slice/crio-0b4c88259c6f089445727cd37a9550f67e174ef0329ac1bc7cd457f12e63521f WatchSource:0}: Error finding container 0b4c88259c6f089445727cd37a9550f67e174ef0329ac1bc7cd457f12e63521f: Status 404 returned error can't find the container with id 0b4c88259c6f089445727cd37a9550f67e174ef0329ac1bc7cd457f12e63521f Jul 07 18:23:56.794475 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:23:56.794444 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2838a03_723b_46e2_8a9f_db1a1625220c.slice/crio-c9c7801506f07cc7a95150b0c45d7aa353594ec160ad979cdacb39750ffb1ec7 WatchSource:0}: Error finding container c9c7801506f07cc7a95150b0c45d7aa353594ec160ad979cdacb39750ffb1ec7: Status 404 returned error can't find the container with id c9c7801506f07cc7a95150b0c45d7aa353594ec160ad979cdacb39750ffb1ec7 Jul 07 18:23:56.802209 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.802187 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fdf557895-96fn2"] Jul 07 18:23:56.824950 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:23:56.824925 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4588b8b8_62be_4d7e_b4a3_8686b47a9d21.slice/crio-b30597331060a0a378b44f4f20893f10a65358a801774bc8e9d2dac5b81c2ea9 WatchSource:0}: Error finding container b30597331060a0a378b44f4f20893f10a65358a801774bc8e9d2dac5b81c2ea9: Status 404 returned error can't find the container with id b30597331060a0a378b44f4f20893f10a65358a801774bc8e9d2dac5b81c2ea9 Jul 07 18:23:56.842852 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.842823 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b68d7f5b-5hs6w"] Jul 07 18:23:56.847799 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.847781 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth"] Jul 07 18:23:56.855392 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:23:56.855361 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1cd385b_fabe_4053_bdcb_9fc07293d8c8.slice/crio-46de1495a42f773be4237b43530831423b2e15827a262afe565f01256ccfa5f5 WatchSource:0}: Error finding container 46de1495a42f773be4237b43530831423b2e15827a262afe565f01256ccfa5f5: Status 404 returned error can't find the container with id 46de1495a42f773be4237b43530831423b2e15827a262afe565f01256ccfa5f5 Jul 07 18:23:56.891235 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.891113 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bncsv"] Jul 07 18:23:56.894204 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:23:56.894175 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03509627_aac5_42cf_a76b_4b7e499bdb4e.slice/crio-81a6f33db6057ffad7110e55897669f65a2a6b755d7891e8dc01ce5fb4372eb2 WatchSource:0}: Error finding container 81a6f33db6057ffad7110e55897669f65a2a6b755d7891e8dc01ce5fb4372eb2: Status 404 returned error can't find the container with id 81a6f33db6057ffad7110e55897669f65a2a6b755d7891e8dc01ce5fb4372eb2 Jul 07 18:23:56.899235 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:56.899198 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:57.044428 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:57.044354 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:57.177184 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:57.177089 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:57.449743 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:57.449654 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fdf557895-96fn2" event={"ID":"e2838a03-723b-46e2-8a9f-db1a1625220c","Type":"ContainerStarted","Data":"c9c7801506f07cc7a95150b0c45d7aa353594ec160ad979cdacb39750ffb1ec7"} Jul 07 18:23:57.451440 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:57.451411 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" event={"ID":"b1cd385b-fabe-4053-bdcb-9fc07293d8c8","Type":"ContainerStarted","Data":"46de1495a42f773be4237b43530831423b2e15827a262afe565f01256ccfa5f5"} Jul 07 18:23:57.454162 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:57.454133 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sxzvf" event={"ID":"be81195f-a6f0-4154-abde-ccaa50f68346","Type":"ContainerStarted","Data":"0b4c88259c6f089445727cd37a9550f67e174ef0329ac1bc7cd457f12e63521f"} Jul 07 18:23:57.456529 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:57.456502 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pd2qx" event={"ID":"5e5a23b5-a6ca-4693-afc4-489102504294","Type":"ContainerStarted","Data":"21384f62e5a5e0388a1364c195d504a644174aa58fc87cd71b165ba2aabe6936"} Jul 07 18:23:57.458819 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:57.458791 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b68d7f5b-5hs6w" event={"ID":"4588b8b8-62be-4d7e-b4a3-8686b47a9d21","Type":"ContainerStarted","Data":"b30597331060a0a378b44f4f20893f10a65358a801774bc8e9d2dac5b81c2ea9"} Jul 07 18:23:57.465002 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:57.460275 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bncsv" event={"ID":"03509627-aac5-42cf-a76b-4b7e499bdb4e","Type":"ContainerStarted","Data":"81a6f33db6057ffad7110e55897669f65a2a6b755d7891e8dc01ce5fb4372eb2"} Jul 07 18:23:57.821430 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:57.821363 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:59.383558 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:59.383259 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcq8x" Jul 07 18:23:59.656933 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:59.656849 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:59.758895 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:23:59.758865 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:01.204823 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:01.204775 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:01.320861 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:01.320823 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:01.488945 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:01.488913 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:01.574956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:01.574919 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:02.827660 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:02.827612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs\") pod \"network-metrics-daemon-bzlxp\" (UID: \"cde64ccc-edac-45b8-b4d3-b260d022fbf5\") " pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:24:02.828159 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:02.827678 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret\") pod \"global-pull-secret-syncer-98qb4\" (UID: \"e17ddfe4-73b3-45bd-b80f-ed992e7e1b71\") " pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:24:02.830636 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:02.830614 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 18:24:02.831081 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:02.831066 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 18:24:02.840472 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:02.840435 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e17ddfe4-73b3-45bd-b80f-ed992e7e1b71-original-pull-secret\") pod \"global-pull-secret-syncer-98qb4\" (UID: \"e17ddfe4-73b3-45bd-b80f-ed992e7e1b71\") " pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:24:02.840472 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:02.840452 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde64ccc-edac-45b8-b4d3-b260d022fbf5-metrics-certs\") pod \"network-metrics-daemon-bzlxp\" (UID: \"cde64ccc-edac-45b8-b4d3-b260d022fbf5\") " pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:24:03.029658 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:03.029563 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kc7t\" (UniqueName: \"kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t\") pod \"network-check-target-k92pn\" (UID: \"3f672f83-cb85-4c82-91fd-1d47b9c67d59\") " pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:24:03.032363 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:03.032335 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 18:24:03.042956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:03.042930 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 18:24:03.054042 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:03.053994 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kc7t\" (UniqueName: \"kubernetes.io/projected/3f672f83-cb85-4c82-91fd-1d47b9c67d59-kube-api-access-6kc7t\") pod \"network-check-target-k92pn\" (UID: \"3f672f83-cb85-4c82-91fd-1d47b9c67d59\") " pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:24:03.105881 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:03.105839 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-98qb4" Jul 07 18:24:03.114191 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:03.114166 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dxsjk\"" Jul 07 18:24:03.117253 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:03.117231 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wxf5l\"" Jul 07 18:24:03.121205 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:03.121185 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:24:03.125766 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:03.125723 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzlxp" Jul 07 18:24:04.928675 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:04.923996 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k92pn"] Jul 07 18:24:04.944612 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:04.942083 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bzlxp"] Jul 07 18:24:04.982640 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:04.982605 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-98qb4"] Jul 07 18:24:05.486499 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:05.486442 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bncsv" event={"ID":"03509627-aac5-42cf-a76b-4b7e499bdb4e","Type":"ContainerStarted","Data":"c79272fe25ffceb5a5af3dd62acb96d59231478d6111cb9c5865de8ee0d3f28f"} Jul 07 18:24:05.486661 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:05.486515 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bncsv" event={"ID":"03509627-aac5-42cf-a76b-4b7e499bdb4e","Type":"ContainerStarted","Data":"eb6caa47436521c3777d68731af5d1f86a9a110b51bbd276d0535e6e2fc78026"} Jul 07 18:24:05.486727 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:05.486660 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bncsv" Jul 07 18:24:05.489010 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:05.488987 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" event={"ID":"b1cd385b-fabe-4053-bdcb-9fc07293d8c8","Type":"ContainerStarted","Data":"1ecb921d29d3fe3224d1ad314de4a0cb197d461f4d79b9b3e2e44a3d73fef2be"} Jul 07 18:24:05.489153 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:05.489021 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" event={"ID":"b1cd385b-fabe-4053-bdcb-9fc07293d8c8","Type":"ContainerStarted","Data":"364d3ba03875dcce7c4a2151091609b407238b1c5c21bfcebdae693c4fbe7eaf"} Jul 07 18:24:05.489153 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:05.489039 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" event={"ID":"b1cd385b-fabe-4053-bdcb-9fc07293d8c8","Type":"ContainerStarted","Data":"013608174b197488d03d1affbbf7c5042fbbba1646df160c585efdcd1c0fb768"} Jul 07 18:24:05.490610 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:05.490582 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sxzvf" event={"ID":"be81195f-a6f0-4154-abde-ccaa50f68346","Type":"ContainerStarted","Data":"a2dd94596baafd0909d76f52cfabdb9b30b02195cded3462f344e701791a50e5"} Jul 07 18:24:05.492186 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:05.492143 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pd2qx" event={"ID":"5e5a23b5-a6ca-4693-afc4-489102504294","Type":"ContainerStarted","Data":"e58a84552d9e8f47640dc9743f711cae262f4486fb5bd2091187af2a4cacd4ca"} Jul 07 18:24:05.493587 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:05.493463 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bzlxp" event={"ID":"cde64ccc-edac-45b8-b4d3-b260d022fbf5","Type":"ContainerStarted","Data":"0b7fac2e57baa0006a421926efb966e447e7d643dead8cdb68e654cee9208ac8"} Jul 07 18:24:05.495373 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:05.495238 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-98qb4" event={"ID":"e17ddfe4-73b3-45bd-b80f-ed992e7e1b71","Type":"ContainerStarted","Data":"4ccffcc434e44c34e818848e132d92cc2761116b085137a8e374be0f733a5852"} Jul 07 18:24:05.496999 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:05.496972 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b68d7f5b-5hs6w" event={"ID":"4588b8b8-62be-4d7e-b4a3-8686b47a9d21","Type":"ContainerStarted","Data":"4704d38adbb2c33aeb886a53c261129fd01f0861f0ec34571e8be14a09b7adb1"} Jul 07 18:24:05.499102 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:05.499077 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k92pn" event={"ID":"3f672f83-cb85-4c82-91fd-1d47b9c67d59","Type":"ContainerStarted","Data":"39c8fea9140a35f98b7729abc6c94c8eb4496d9b60a214d44e8a270cd2b844d5"} Jul 07 18:24:05.544218 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:05.544156 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sxzvf" podStartSLOduration=1.600265964 podStartE2EDuration="9.544134244s" totalImagesPullingTime="7.94386828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:05.542715389 +0000 UTC m=+68.926508809" watchObservedRunningTime="2026-07-07 18:24:05.544134244 +0000 UTC m=+68.927927640" Jul 07 18:24:05.544394 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:05.544286 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bncsv" podStartSLOduration=2.065084827 podStartE2EDuration="9.544278383s" totalImagesPullingTime="7.479193556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:05.517832939 +0000 UTC m=+68.901626335" watchObservedRunningTime="2026-07-07 18:24:05.544278383 +0000 UTC m=+68.928071757" Jul 07 18:24:05.565772 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:05.565612 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b68d7f5b-5hs6w" podStartSLOduration=1.676160699 podStartE2EDuration="9.56559169s" totalImagesPullingTime="7.889430991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:05.564659308 +0000 UTC m=+68.948452704" watchObservedRunningTime="2026-07-07 18:24:05.56559169 +0000 UTC m=+68.949385086" Jul 07 18:24:05.601027 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:05.598578 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85995f898d-wtbth" podStartSLOduration=1.706034998 podStartE2EDuration="9.598556794s" totalImagesPullingTime="7.892521796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:05.597132921 +0000 UTC m=+68.980926313" watchObservedRunningTime="2026-07-07 18:24:05.598556794 +0000 UTC m=+68.982350190" Jul 07 18:24:05.825804 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:05.825708 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:06.035734 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:06.035647 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:06.508578 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:06.507730 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pd2qx" event={"ID":"5e5a23b5-a6ca-4693-afc4-489102504294","Type":"ContainerStarted","Data":"e9b75826cb455ac9b8056c4b735888cfff7afc7bca8d3eb52bf250c2e82d16ef"} Jul 07 18:24:06.529130 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:06.529057 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:06.637650 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:06.637618 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:07.514508 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:07.514468 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bzlxp" event={"ID":"cde64ccc-edac-45b8-b4d3-b260d022fbf5","Type":"ContainerStarted","Data":"c525574b170d0bf11baa1cbbb53ce36dbbef4ebe02dd9dfcc4ae6b76567e3225"} Jul 07 18:24:07.514974 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:07.514539 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bzlxp" event={"ID":"cde64ccc-edac-45b8-b4d3-b260d022fbf5","Type":"ContainerStarted","Data":"38f73e8900d236faf7ece6e09f9ab319ba1c8092ef1b7bb9d101a9bbf047fb08"} Jul 07 18:24:07.538886 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:07.538708 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bzlxp" podStartSLOduration=68.880478814 podStartE2EDuration="1m10.538686536s" totalImagesPullingTime="1.658207722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:07.538398338 +0000 UTC m=+70.922191736" watchObservedRunningTime="2026-07-07 18:24:07.538686536 +0000 UTC m=+70.922479936" Jul 07 18:24:07.697743 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:07.697660 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:07.797759 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:07.797596 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:08.393279 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.393240 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:08.622398 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.622369 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:08.732917 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.732719 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gxdbb"] Jul 07 18:24:08.739365 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.739332 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.743238 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.743210 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 18:24:08.743824 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.743800 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 18:24:08.743824 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.743809 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 18:24:08.744023 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.743872 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 18:24:08.744023 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.743916 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 18:24:08.744023 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.743948 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 18:24:08.744278 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.744261 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-j2kzd\"" Jul 07 18:24:08.766532 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.766497 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:08.782254 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.782216 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d0acf142-00b0-44ca-a46d-7d8e3f86f841-root\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.782254 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.782245 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0acf142-00b0-44ca-a46d-7d8e3f86f841-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.782484 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.782369 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d0acf142-00b0-44ca-a46d-7d8e3f86f841-node-exporter-tls\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.782484 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.782418 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d0acf142-00b0-44ca-a46d-7d8e3f86f841-node-exporter-textfile\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.782484 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.782449 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d0acf142-00b0-44ca-a46d-7d8e3f86f841-node-exporter-accelerators-collector-config\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.782672 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.782516 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d0acf142-00b0-44ca-a46d-7d8e3f86f841-node-exporter-wtmp\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.782672 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.782558 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgfcf\" (UniqueName: \"kubernetes.io/projected/d0acf142-00b0-44ca-a46d-7d8e3f86f841-kube-api-access-rgfcf\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.782672 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.782588 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0acf142-00b0-44ca-a46d-7d8e3f86f841-metrics-client-ca\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.782672 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.782644 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0acf142-00b0-44ca-a46d-7d8e3f86f841-sys\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.842128 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.842085 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:08.883300 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.883267 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d0acf142-00b0-44ca-a46d-7d8e3f86f841-node-exporter-wtmp\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.883503 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.883310 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgfcf\" (UniqueName: \"kubernetes.io/projected/d0acf142-00b0-44ca-a46d-7d8e3f86f841-kube-api-access-rgfcf\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.883503 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.883360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0acf142-00b0-44ca-a46d-7d8e3f86f841-metrics-client-ca\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.883503 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.883392 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0acf142-00b0-44ca-a46d-7d8e3f86f841-sys\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.883503 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.883417 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d0acf142-00b0-44ca-a46d-7d8e3f86f841-root\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.883503 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.883437 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0acf142-00b0-44ca-a46d-7d8e3f86f841-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.883503 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.883488 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d0acf142-00b0-44ca-a46d-7d8e3f86f841-node-exporter-wtmp\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.883503 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.883499 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d0acf142-00b0-44ca-a46d-7d8e3f86f841-node-exporter-tls\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.883825 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.883551 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d0acf142-00b0-44ca-a46d-7d8e3f86f841-node-exporter-textfile\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.883825 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.883581 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d0acf142-00b0-44ca-a46d-7d8e3f86f841-node-exporter-accelerators-collector-config\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.884130 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.884107 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d0acf142-00b0-44ca-a46d-7d8e3f86f841-root\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.884225 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.884147 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d0acf142-00b0-44ca-a46d-7d8e3f86f841-node-exporter-accelerators-collector-config\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.884225 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.884168 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0acf142-00b0-44ca-a46d-7d8e3f86f841-sys\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.884225 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.884150 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0acf142-00b0-44ca-a46d-7d8e3f86f841-metrics-client-ca\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.884413 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.884400 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d0acf142-00b0-44ca-a46d-7d8e3f86f841-node-exporter-textfile\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.886530 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.886505 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0acf142-00b0-44ca-a46d-7d8e3f86f841-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.886642 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.886572 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d0acf142-00b0-44ca-a46d-7d8e3f86f841-node-exporter-tls\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.893434 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.893411 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgfcf\" (UniqueName: \"kubernetes.io/projected/d0acf142-00b0-44ca-a46d-7d8e3f86f841-kube-api-access-rgfcf\") pod \"node-exporter-gxdbb\" (UID: \"d0acf142-00b0-44ca-a46d-7d8e3f86f841\") " pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:08.944165 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:08.944126 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:09.046238 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:09.046156 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:09.051101 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:09.051077 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gxdbb" Jul 07 18:24:09.190740 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:09.190654 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:09.680175 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:09.680115 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:09.790088 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:09.790044 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:09.876305 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:09.876272 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:11.021020 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:24:11.020988 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0acf142_00b0_44ca_a46d_7d8e3f86f841.slice/crio-dc5af0fdfaeeaf4eb0e09c495bc9742f012896cac0d23c221058729fa5d6ebd1 WatchSource:0}: Error finding container dc5af0fdfaeeaf4eb0e09c495bc9742f012896cac0d23c221058729fa5d6ebd1: Status 404 returned error can't find the container with id dc5af0fdfaeeaf4eb0e09c495bc9742f012896cac0d23c221058729fa5d6ebd1 Jul 07 18:24:11.534007 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:11.533970 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pd2qx" event={"ID":"5e5a23b5-a6ca-4693-afc4-489102504294","Type":"ContainerStarted","Data":"37e4a07b34f7678c1268fee7923db62eeb7e7202a2bb12f2ec42f7a5aec7dd9e"} Jul 07 18:24:11.535444 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:11.535412 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-98qb4" event={"ID":"e17ddfe4-73b3-45bd-b80f-ed992e7e1b71","Type":"ContainerStarted","Data":"03b411590d96286cd17ece12cce879a98678f20c80430a0d8aa6276fd2ac87ae"} Jul 07 18:24:11.536901 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:11.536876 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k92pn" event={"ID":"3f672f83-cb85-4c82-91fd-1d47b9c67d59","Type":"ContainerStarted","Data":"f9a64daf63e62522b2edcba5d57b3778834b82c0f2957207d01541a6a3a2e1a3"} Jul 07 18:24:11.536999 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:11.536934 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:24:11.538441 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:11.538414 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gxdbb" event={"ID":"d0acf142-00b0-44ca-a46d-7d8e3f86f841","Type":"ContainerStarted","Data":"dc5af0fdfaeeaf4eb0e09c495bc9742f012896cac0d23c221058729fa5d6ebd1"} Jul 07 18:24:11.554765 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:11.554716 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pd2qx" podStartSLOduration=1.35698849 podStartE2EDuration="15.554700945s" totalImagesPullingTime="14.197712455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:23:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:11.55342691 +0000 UTC m=+74.937220302" watchObservedRunningTime="2026-07-07 18:24:11.554700945 +0000 UTC m=+74.938494338" Jul 07 18:24:11.594919 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:11.594851 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-98qb4" podStartSLOduration=68.562517151 podStartE2EDuration="1m14.594831077s" totalImagesPullingTime="6.032313926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:11.571938182 +0000 UTC m=+74.955731577" watchObservedRunningTime="2026-07-07 18:24:11.594831077 +0000 UTC m=+74.978624474" Jul 07 18:24:11.595093 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:11.595012 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-k92pn" podStartSLOduration=68.50776141 podStartE2EDuration="1m14.595007066s" totalImagesPullingTime="6.087245656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:11.59407525 +0000 UTC m=+74.977868640" watchObservedRunningTime="2026-07-07 18:24:11.595007066 +0000 UTC m=+74.978800462" Jul 07 18:24:11.731137 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:11.731106 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:11.795520 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:11.795446 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:11.864098 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:11.864049 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:12.186351 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:12.186255 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:13.412794 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:13.412755 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:13.541304 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:13.541270 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:13.547618 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:13.547589 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fdf557895-96fn2" event={"ID":"e2838a03-723b-46e2-8a9f-db1a1625220c","Type":"ContainerStarted","Data":"6f8e83a1c23c30276c230700ae0439ca1f0dc43006b7f86c9479e701dc51a9a1"} Jul 07 18:24:13.547770 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:13.547653 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fdf557895-96fn2" Jul 07 18:24:13.549261 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:13.549236 2578 generic.go:356] "Generic (PLEG): container finished" podID="d0acf142-00b0-44ca-a46d-7d8e3f86f841" containerID="a59b34625cbc047a93d4c69cebd453898fe748dc94bf455bff39f5e02bdd7d85" exitCode=0 Jul 07 18:24:13.549401 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:13.549293 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gxdbb" event={"ID":"d0acf142-00b0-44ca-a46d-7d8e3f86f841","Type":"ContainerDied","Data":"a59b34625cbc047a93d4c69cebd453898fe748dc94bf455bff39f5e02bdd7d85"} Jul 07 18:24:13.549678 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:13.549660 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fdf557895-96fn2" Jul 07 18:24:13.568789 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:13.568727 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fdf557895-96fn2" podStartSLOduration=1.043017063 podStartE2EDuration="17.568710532s" totalImagesPullingTime="16.525693469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:13.567271779 +0000 UTC m=+76.951065196" watchObservedRunningTime="2026-07-07 18:24:13.568710532 +0000 UTC m=+76.952503928" Jul 07 18:24:14.554963 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:14.554919 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gxdbb" event={"ID":"d0acf142-00b0-44ca-a46d-7d8e3f86f841","Type":"ContainerStarted","Data":"b3a465988cafbb128ad84dd0ad337c9bc7fd7252c7831805f4fb830ae49baf2b"} Jul 07 18:24:14.555458 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:14.554987 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gxdbb" event={"ID":"d0acf142-00b0-44ca-a46d-7d8e3f86f841","Type":"ContainerStarted","Data":"77e2a6ba01b6ff2a50f9b7b508d2cb06368f447398528e5bd9dc980baac99a8e"} Jul 07 18:24:14.831410 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:14.831279 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:15.042754 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:15.042723 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:15.198068 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:15.197973 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:15.308722 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:15.308686 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:15.491850 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:15.491816 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:15.511274 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:15.511246 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bncsv" Jul 07 18:24:15.534183 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:15.534125 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gxdbb" podStartSLOduration=5.174538354 podStartE2EDuration="7.534105576s" totalImagesPullingTime="2.287375618s" totalInitContainerRuntime="72.191604ms" isStatefulPod=true podCreationTimestamp="2026-07-07 18:24:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:14.600629657 +0000 UTC m=+77.984423053" watchObservedRunningTime="2026-07-07 18:24:15.534105576 +0000 UTC m=+78.917899042" Jul 07 18:24:15.648371 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:15.648335 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:15.904052 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:15.903966 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:16.056210 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:16.056177 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:16.214736 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:16.214625 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:16.340577 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:16.340544 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:16.491704 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:16.491648 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:16.814588 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:16.814469 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:17.296455 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:17.296380 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:17.549449 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:17.549341 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:17.740505 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:17.740468 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:18.513755 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:18.513722 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:18.615653 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:18.615618 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:18.732634 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:18.732579 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:18.836306 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:18.836206 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:19.800019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:19.799984 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:19.942050 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:19.942012 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:21.572421 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:21.572387 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:21.685890 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:21.685857 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:24.300511 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:24.300474 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:24.419347 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:24.419288 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:24.858439 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:24.858405 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:25.043596 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:25.043563 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:26.297269 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:26.297223 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:26.404857 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:26.404827 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:28.069469 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.069433 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bbbf5bc7c-6mjz5"] Jul 07 18:24:28.072143 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.072125 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.076881 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.076860 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 18:24:28.079805 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.079781 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 18:24:28.080607 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.080588 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 18:24:28.080918 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.080882 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-sh5tt\"" Jul 07 18:24:28.081013 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.080927 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 18:24:28.081013 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.080942 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 18:24:28.081013 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.080927 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 18:24:28.082952 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.082935 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 18:24:28.086997 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.086978 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 18:24:28.107487 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.107448 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bbbf5bc7c-6mjz5"] Jul 07 18:24:28.230273 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.230230 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kl55\" (UniqueName: \"kubernetes.io/projected/a46c34f8-93ac-47be-92d1-adaba5966d6a-kube-api-access-5kl55\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.230273 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.230277 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-console-config\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.230597 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.230309 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-oauth-serving-cert\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.230597 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.230363 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-trusted-ca-bundle\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.230597 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.230423 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a46c34f8-93ac-47be-92d1-adaba5966d6a-console-oauth-config\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.230597 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.230443 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a46c34f8-93ac-47be-92d1-adaba5966d6a-console-serving-cert\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.230597 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.230456 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-service-ca\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.331138 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.331049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kl55\" (UniqueName: \"kubernetes.io/projected/a46c34f8-93ac-47be-92d1-adaba5966d6a-kube-api-access-5kl55\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.331138 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.331096 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-console-config\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.331138 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.331134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-oauth-serving-cert\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.331451 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.331167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-trusted-ca-bundle\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.331451 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.331195 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a46c34f8-93ac-47be-92d1-adaba5966d6a-console-oauth-config\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.331451 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.331220 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a46c34f8-93ac-47be-92d1-adaba5966d6a-console-serving-cert\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.331451 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.331241 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-service-ca\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.331946 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.331921 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-console-config\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.331946 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.331941 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-oauth-serving-cert\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.332151 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.332128 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-service-ca\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.332303 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.332279 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-trusted-ca-bundle\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.333781 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.333757 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a46c34f8-93ac-47be-92d1-adaba5966d6a-console-serving-cert\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.333912 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.333891 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a46c34f8-93ac-47be-92d1-adaba5966d6a-console-oauth-config\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.356847 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.356816 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kl55\" (UniqueName: \"kubernetes.io/projected/a46c34f8-93ac-47be-92d1-adaba5966d6a-kube-api-access-5kl55\") pod \"console-6bbbf5bc7c-6mjz5\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.381834 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.381794 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:28.547052 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.547017 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bbbf5bc7c-6mjz5"] Jul 07 18:24:28.550260 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:24:28.550221 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46c34f8_93ac_47be_92d1_adaba5966d6a.slice/crio-1412ceba10d5837afe49d22cc1ffc5be170c300c9484438bba68f2ef869314c9 WatchSource:0}: Error finding container 1412ceba10d5837afe49d22cc1ffc5be170c300c9484438bba68f2ef869314c9: Status 404 returned error can't find the container with id 1412ceba10d5837afe49d22cc1ffc5be170c300c9484438bba68f2ef869314c9 Jul 07 18:24:28.606266 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.606184 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bbbf5bc7c-6mjz5" event={"ID":"a46c34f8-93ac-47be-92d1-adaba5966d6a","Type":"ContainerStarted","Data":"1412ceba10d5837afe49d22cc1ffc5be170c300c9484438bba68f2ef869314c9"} Jul 07 18:24:28.781218 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.781187 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:28.965042 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:28.964953 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:29.081541 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:29.081511 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:32.623589 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:32.623552 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bbbf5bc7c-6mjz5" event={"ID":"a46c34f8-93ac-47be-92d1-adaba5966d6a","Type":"ContainerStarted","Data":"0658da50faa9b9c137a30e0f7fefeaa94dd711ed3b0357de46e9b04aa63451dd"} Jul 07 18:24:32.647130 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:32.647076 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bbbf5bc7c-6mjz5" podStartSLOduration=1.426447017 podStartE2EDuration="4.647058823s" totalImagesPullingTime="3.220611806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:24:32.645889409 +0000 UTC m=+96.029682805" watchObservedRunningTime="2026-07-07 18:24:32.647058823 +0000 UTC m=+96.030852221" Jul 07 18:24:33.285298 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:33.285263 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:33.408434 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:33.408400 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:36.639576 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:36.639534 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:36.828693 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:36.828662 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:37.255251 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:37.255214 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:37.336396 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:37.336361 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:38.382109 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:38.382067 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:38.382512 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:38.382144 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:38.386889 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:38.386866 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:38.646510 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:38.646437 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:24:42.545631 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:42.545600 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k92pn" Jul 07 18:24:43.449783 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:43.449735 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:43.629174 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:43.629137 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:51.562872 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:51.562831 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:51.661190 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:24:51.661155 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:28.352209 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:25:28.352173 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:28.623821 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:25:28.623735 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:37.986136 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:25:37.986099 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:38.184889 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:25:38.184852 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:49.185022 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:25:49.184944 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bbbf5bc7c-6mjz5"] Jul 07 18:25:49.357430 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:25:49.357397 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:49.457144 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:25:49.457055 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:49.673272 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:25:49.673238 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:50.820764 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:25:50.820723 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:14.205834 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.205793 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6bbbf5bc7c-6mjz5" podUID="a46c34f8-93ac-47be-92d1-adaba5966d6a" containerName="console" containerID="cri-o://0658da50faa9b9c137a30e0f7fefeaa94dd711ed3b0357de46e9b04aa63451dd" gracePeriod=15 Jul 07 18:26:14.436148 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.436124 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:26:14.509959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.509927 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a46c34f8-93ac-47be-92d1-adaba5966d6a-console-oauth-config\") pod \"a46c34f8-93ac-47be-92d1-adaba5966d6a\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " Jul 07 18:26:14.510152 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.509967 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a46c34f8-93ac-47be-92d1-adaba5966d6a-console-serving-cert\") pod \"a46c34f8-93ac-47be-92d1-adaba5966d6a\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " Jul 07 18:26:14.510152 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.509991 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-oauth-serving-cert\") pod \"a46c34f8-93ac-47be-92d1-adaba5966d6a\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " Jul 07 18:26:14.510152 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.510010 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kl55\" (UniqueName: \"kubernetes.io/projected/a46c34f8-93ac-47be-92d1-adaba5966d6a-kube-api-access-5kl55\") pod \"a46c34f8-93ac-47be-92d1-adaba5966d6a\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " Jul 07 18:26:14.510152 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.510058 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-trusted-ca-bundle\") pod \"a46c34f8-93ac-47be-92d1-adaba5966d6a\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " Jul 07 18:26:14.510152 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.510086 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-service-ca\") pod \"a46c34f8-93ac-47be-92d1-adaba5966d6a\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " Jul 07 18:26:14.510152 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.510113 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-console-config\") pod \"a46c34f8-93ac-47be-92d1-adaba5966d6a\" (UID: \"a46c34f8-93ac-47be-92d1-adaba5966d6a\") " Jul 07 18:26:14.510604 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.510572 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a46c34f8-93ac-47be-92d1-adaba5966d6a" (UID: "a46c34f8-93ac-47be-92d1-adaba5966d6a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:26:14.510604 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.510588 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-console-config" (OuterVolumeSpecName: "console-config") pod "a46c34f8-93ac-47be-92d1-adaba5966d6a" (UID: "a46c34f8-93ac-47be-92d1-adaba5966d6a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:26:14.510758 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.510637 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a46c34f8-93ac-47be-92d1-adaba5966d6a" (UID: "a46c34f8-93ac-47be-92d1-adaba5966d6a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:26:14.510758 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.510632 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-service-ca" (OuterVolumeSpecName: "service-ca") pod "a46c34f8-93ac-47be-92d1-adaba5966d6a" (UID: "a46c34f8-93ac-47be-92d1-adaba5966d6a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:26:14.512334 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.512300 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a46c34f8-93ac-47be-92d1-adaba5966d6a-kube-api-access-5kl55" (OuterVolumeSpecName: "kube-api-access-5kl55") pod "a46c34f8-93ac-47be-92d1-adaba5966d6a" (UID: "a46c34f8-93ac-47be-92d1-adaba5966d6a"). InnerVolumeSpecName "kube-api-access-5kl55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:26:14.512605 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.512578 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a46c34f8-93ac-47be-92d1-adaba5966d6a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a46c34f8-93ac-47be-92d1-adaba5966d6a" (UID: "a46c34f8-93ac-47be-92d1-adaba5966d6a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:26:14.512605 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.512594 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a46c34f8-93ac-47be-92d1-adaba5966d6a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a46c34f8-93ac-47be-92d1-adaba5966d6a" (UID: "a46c34f8-93ac-47be-92d1-adaba5966d6a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:26:14.610624 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.610586 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kl55\" (UniqueName: \"kubernetes.io/projected/a46c34f8-93ac-47be-92d1-adaba5966d6a-kube-api-access-5kl55\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:26:14.610624 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.610616 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-trusted-ca-bundle\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:26:14.610624 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.610626 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-service-ca\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:26:14.610624 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.610635 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-console-config\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:26:14.610897 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.610642 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a46c34f8-93ac-47be-92d1-adaba5966d6a-console-oauth-config\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:26:14.610897 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.610651 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a46c34f8-93ac-47be-92d1-adaba5966d6a-console-serving-cert\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:26:14.610897 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.610659 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a46c34f8-93ac-47be-92d1-adaba5966d6a-oauth-serving-cert\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:26:14.953360 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.953249 2578 generic.go:356] "Generic (PLEG): container finished" podID="a46c34f8-93ac-47be-92d1-adaba5966d6a" containerID="0658da50faa9b9c137a30e0f7fefeaa94dd711ed3b0357de46e9b04aa63451dd" exitCode=0 Jul 07 18:26:14.953360 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.953288 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bbbf5bc7c-6mjz5" event={"ID":"a46c34f8-93ac-47be-92d1-adaba5966d6a","Type":"ContainerDied","Data":"0658da50faa9b9c137a30e0f7fefeaa94dd711ed3b0357de46e9b04aa63451dd"} Jul 07 18:26:14.953360 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.953302 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bbbf5bc7c-6mjz5" Jul 07 18:26:14.953360 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.953348 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bbbf5bc7c-6mjz5" event={"ID":"a46c34f8-93ac-47be-92d1-adaba5966d6a","Type":"ContainerDied","Data":"1412ceba10d5837afe49d22cc1ffc5be170c300c9484438bba68f2ef869314c9"} Jul 07 18:26:14.953699 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.953375 2578 scope.go:117] "RemoveContainer" containerID="0658da50faa9b9c137a30e0f7fefeaa94dd711ed3b0357de46e9b04aa63451dd" Jul 07 18:26:14.962021 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.962002 2578 scope.go:117] "RemoveContainer" containerID="0658da50faa9b9c137a30e0f7fefeaa94dd711ed3b0357de46e9b04aa63451dd" Jul 07 18:26:14.962303 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:26:14.962278 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0658da50faa9b9c137a30e0f7fefeaa94dd711ed3b0357de46e9b04aa63451dd\": container with ID starting with 0658da50faa9b9c137a30e0f7fefeaa94dd711ed3b0357de46e9b04aa63451dd not found: ID does not exist" containerID="0658da50faa9b9c137a30e0f7fefeaa94dd711ed3b0357de46e9b04aa63451dd" Jul 07 18:26:14.962422 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.962309 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0658da50faa9b9c137a30e0f7fefeaa94dd711ed3b0357de46e9b04aa63451dd"} err="failed to get container status \"0658da50faa9b9c137a30e0f7fefeaa94dd711ed3b0357de46e9b04aa63451dd\": rpc error: code = NotFound desc = could not find container \"0658da50faa9b9c137a30e0f7fefeaa94dd711ed3b0357de46e9b04aa63451dd\": container with ID starting with 0658da50faa9b9c137a30e0f7fefeaa94dd711ed3b0357de46e9b04aa63451dd not found: ID does not exist" Jul 07 18:26:14.973864 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.973827 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bbbf5bc7c-6mjz5"] Jul 07 18:26:14.978121 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:14.978096 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bbbf5bc7c-6mjz5"] Jul 07 18:26:15.195409 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:15.195380 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a46c34f8-93ac-47be-92d1-adaba5966d6a" path="/var/lib/kubelet/pods/a46c34f8-93ac-47be-92d1-adaba5966d6a/volumes" Jul 07 18:26:20.699379 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:20.699340 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:20.791589 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:20.791560 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:43.507666 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:43.507629 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:43.580478 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:43.580441 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:55.458366 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:55.458315 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:55.515665 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:26:55.515637 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:27:06.574008 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:27:06.573974 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:27:06.669524 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:27:06.669488 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:27:06.779241 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:27:06.779213 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:27:14.828878 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:27:14.828845 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:27:14.930157 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:27:14.930123 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:27:25.687248 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:27:25.687172 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:27:25.788439 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:27:25.788399 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:27:51.996580 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:27:51.996538 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:27:52.125419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:27:52.125385 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:27:55.032667 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:27:55.032636 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:27:57.030406 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:27:57.030376 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 18:27:57.032176 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:27:57.032153 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 18:27:57.034739 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:27:57.034725 2578 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 18:28:24.929736 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:24.929694 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:25.246792 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:25.246740 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:28.713641 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:28.713610 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:28.819855 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:28.819813 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:30.222235 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:30.222185 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:30.447621 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:30.447580 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:30.770168 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:30.770117 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:31.432946 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:31.432914 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mqczn"] Jul 07 18:28:31.433312 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:31.433128 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a46c34f8-93ac-47be-92d1-adaba5966d6a" containerName="console" Jul 07 18:28:31.433312 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:31.433139 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a46c34f8-93ac-47be-92d1-adaba5966d6a" containerName="console" Jul 07 18:28:31.433312 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:31.433175 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a46c34f8-93ac-47be-92d1-adaba5966d6a" containerName="console" Jul 07 18:28:31.435315 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:31.435300 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-mqczn" Jul 07 18:28:31.439919 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:31.439902 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 18:28:31.440824 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:31.440806 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 18:28:31.441088 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:31.441069 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-9tvgh\"" Jul 07 18:28:31.539188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:31.539150 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgs2l\" (UniqueName: \"kubernetes.io/projected/0b87d59b-48b0-406a-9218-58c5ff5b7088-kube-api-access-dgs2l\") pod \"cert-manager-webhook-6486ffd997-mqczn\" (UID: \"0b87d59b-48b0-406a-9218-58c5ff5b7088\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mqczn" Jul 07 18:28:31.539384 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:31.539197 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b87d59b-48b0-406a-9218-58c5ff5b7088-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mqczn\" (UID: \"0b87d59b-48b0-406a-9218-58c5ff5b7088\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mqczn" Jul 07 18:28:31.555552 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:31.555524 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mqczn"] Jul 07 18:28:31.639831 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:31.639792 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgs2l\" (UniqueName: \"kubernetes.io/projected/0b87d59b-48b0-406a-9218-58c5ff5b7088-kube-api-access-dgs2l\") pod \"cert-manager-webhook-6486ffd997-mqczn\" (UID: \"0b87d59b-48b0-406a-9218-58c5ff5b7088\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mqczn" Jul 07 18:28:31.640035 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:31.639846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b87d59b-48b0-406a-9218-58c5ff5b7088-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mqczn\" (UID: \"0b87d59b-48b0-406a-9218-58c5ff5b7088\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mqczn" Jul 07 18:28:31.650963 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:31.650932 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgs2l\" (UniqueName: \"kubernetes.io/projected/0b87d59b-48b0-406a-9218-58c5ff5b7088-kube-api-access-dgs2l\") pod \"cert-manager-webhook-6486ffd997-mqczn\" (UID: \"0b87d59b-48b0-406a-9218-58c5ff5b7088\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mqczn" Jul 07 18:28:31.652773 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:31.652749 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b87d59b-48b0-406a-9218-58c5ff5b7088-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mqczn\" (UID: \"0b87d59b-48b0-406a-9218-58c5ff5b7088\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mqczn" Jul 07 18:28:31.744017 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:31.743983 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-mqczn" Jul 07 18:28:31.869531 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:31.869499 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mqczn"] Jul 07 18:28:31.872555 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:28:31.872526 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b87d59b_48b0_406a_9218_58c5ff5b7088.slice/crio-773bc0b2ed115151ea6041ca2030997ea6dae184dadeee8d6beef247b5db5aea WatchSource:0}: Error finding container 773bc0b2ed115151ea6041ca2030997ea6dae184dadeee8d6beef247b5db5aea: Status 404 returned error can't find the container with id 773bc0b2ed115151ea6041ca2030997ea6dae184dadeee8d6beef247b5db5aea Jul 07 18:28:31.874409 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:31.874392 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 18:28:32.388459 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:32.388429 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-mqczn" event={"ID":"0b87d59b-48b0-406a-9218-58c5ff5b7088","Type":"ContainerStarted","Data":"773bc0b2ed115151ea6041ca2030997ea6dae184dadeee8d6beef247b5db5aea"} Jul 07 18:28:32.417670 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:32.417645 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:32.783728 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:32.783689 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:35.401239 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:35.401204 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-mqczn" event={"ID":"0b87d59b-48b0-406a-9218-58c5ff5b7088","Type":"ContainerStarted","Data":"c324c5f08bdfb39c874bfa13e2bd220adb7ba910e7e62f2e3df33b2a75c1c775"} Jul 07 18:28:35.401654 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:35.401362 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-mqczn" Jul 07 18:28:35.418912 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:35.418856 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-mqczn" podStartSLOduration=1.604953716 podStartE2EDuration="4.418836887s" totalImagesPullingTime="2.813883171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:28:35.417548568 +0000 UTC m=+338.801341965" watchObservedRunningTime="2026-07-07 18:28:35.418836887 +0000 UTC m=+338.802630282" Jul 07 18:28:35.502650 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:35.502605 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:35.573277 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:35.573247 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:40.387315 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:40.387286 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:40.499665 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:40.499634 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:40.598635 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:40.598607 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:40.698349 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:40.698271 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:41.406572 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:41.406540 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-mqczn" Jul 07 18:28:44.975647 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:44.975611 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:47.675518 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:47.675436 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:47.774022 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:47.773991 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:49.686718 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:28:49.686684 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:01.728829 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:01.728798 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:01.840926 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:01.840892 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:02.332828 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.332795 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:02.594311 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.594228 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:02.696664 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.696632 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:02.735575 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.735531 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq"] Jul 07 18:29:02.738578 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.738562 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" Jul 07 18:29:02.741070 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.741032 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 18:29:02.741202 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.741077 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 18:29:02.741202 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.741077 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 18:29:02.741345 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.741330 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 18:29:02.741995 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.741976 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-llbm9\"" Jul 07 18:29:02.758444 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.758415 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq"] Jul 07 18:29:02.862855 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.862757 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1b4458c2-e0f3-4be4-9691-58e3f4f42d38-manifests\") pod \"opendatahub-operator-controller-manager-7595c859c8-vk6nq\" (UID: \"1b4458c2-e0f3-4be4-9691-58e3f4f42d38\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" Jul 07 18:29:02.862855 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.862799 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b4458c2-e0f3-4be4-9691-58e3f4f42d38-webhook-cert\") pod \"opendatahub-operator-controller-manager-7595c859c8-vk6nq\" (UID: \"1b4458c2-e0f3-4be4-9691-58e3f4f42d38\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" Jul 07 18:29:02.862855 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.862819 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b4458c2-e0f3-4be4-9691-58e3f4f42d38-tmp\") pod \"opendatahub-operator-controller-manager-7595c859c8-vk6nq\" (UID: \"1b4458c2-e0f3-4be4-9691-58e3f4f42d38\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" Jul 07 18:29:02.863084 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.862863 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1b4458c2-e0f3-4be4-9691-58e3f4f42d38-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7595c859c8-vk6nq\" (UID: \"1b4458c2-e0f3-4be4-9691-58e3f4f42d38\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" Jul 07 18:29:02.863084 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.862915 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mlbw\" (UniqueName: \"kubernetes.io/projected/1b4458c2-e0f3-4be4-9691-58e3f4f42d38-kube-api-access-2mlbw\") pod \"opendatahub-operator-controller-manager-7595c859c8-vk6nq\" (UID: \"1b4458c2-e0f3-4be4-9691-58e3f4f42d38\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" Jul 07 18:29:02.928569 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.928528 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:02.964252 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.964226 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1b4458c2-e0f3-4be4-9691-58e3f4f42d38-manifests\") pod \"opendatahub-operator-controller-manager-7595c859c8-vk6nq\" (UID: \"1b4458c2-e0f3-4be4-9691-58e3f4f42d38\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" Jul 07 18:29:02.964422 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.964257 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b4458c2-e0f3-4be4-9691-58e3f4f42d38-webhook-cert\") pod \"opendatahub-operator-controller-manager-7595c859c8-vk6nq\" (UID: \"1b4458c2-e0f3-4be4-9691-58e3f4f42d38\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" Jul 07 18:29:02.964422 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.964295 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b4458c2-e0f3-4be4-9691-58e3f4f42d38-tmp\") pod \"opendatahub-operator-controller-manager-7595c859c8-vk6nq\" (UID: \"1b4458c2-e0f3-4be4-9691-58e3f4f42d38\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" Jul 07 18:29:02.964422 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.964338 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1b4458c2-e0f3-4be4-9691-58e3f4f42d38-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7595c859c8-vk6nq\" (UID: \"1b4458c2-e0f3-4be4-9691-58e3f4f42d38\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" Jul 07 18:29:02.964422 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.964379 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mlbw\" (UniqueName: \"kubernetes.io/projected/1b4458c2-e0f3-4be4-9691-58e3f4f42d38-kube-api-access-2mlbw\") pod \"opendatahub-operator-controller-manager-7595c859c8-vk6nq\" (UID: \"1b4458c2-e0f3-4be4-9691-58e3f4f42d38\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" Jul 07 18:29:02.964717 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.964693 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1b4458c2-e0f3-4be4-9691-58e3f4f42d38-manifests\") pod \"opendatahub-operator-controller-manager-7595c859c8-vk6nq\" (UID: \"1b4458c2-e0f3-4be4-9691-58e3f4f42d38\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" Jul 07 18:29:02.966707 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.966685 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b4458c2-e0f3-4be4-9691-58e3f4f42d38-tmp\") pod \"opendatahub-operator-controller-manager-7595c859c8-vk6nq\" (UID: \"1b4458c2-e0f3-4be4-9691-58e3f4f42d38\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" Jul 07 18:29:02.966825 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.966772 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b4458c2-e0f3-4be4-9691-58e3f4f42d38-webhook-cert\") pod \"opendatahub-operator-controller-manager-7595c859c8-vk6nq\" (UID: \"1b4458c2-e0f3-4be4-9691-58e3f4f42d38\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" Jul 07 18:29:02.966825 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.966807 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1b4458c2-e0f3-4be4-9691-58e3f4f42d38-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7595c859c8-vk6nq\" (UID: \"1b4458c2-e0f3-4be4-9691-58e3f4f42d38\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" Jul 07 18:29:02.977714 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:02.977685 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mlbw\" (UniqueName: \"kubernetes.io/projected/1b4458c2-e0f3-4be4-9691-58e3f4f42d38-kube-api-access-2mlbw\") pod \"opendatahub-operator-controller-manager-7595c859c8-vk6nq\" (UID: \"1b4458c2-e0f3-4be4-9691-58e3f4f42d38\") " pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" Jul 07 18:29:03.029727 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:03.029693 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:03.048655 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:03.048629 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" Jul 07 18:29:03.131544 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:03.131509 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:03.181947 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:03.181855 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq"] Jul 07 18:29:03.184619 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:29:03.184592 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4458c2_e0f3_4be4_9691_58e3f4f42d38.slice/crio-5139aee9bfa74a70df07a4092176b275c8ace7b4fa898ab3eada980ce8fcd221 WatchSource:0}: Error finding container 5139aee9bfa74a70df07a4092176b275c8ace7b4fa898ab3eada980ce8fcd221: Status 404 returned error can't find the container with id 5139aee9bfa74a70df07a4092176b275c8ace7b4fa898ab3eada980ce8fcd221 Jul 07 18:29:03.218041 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:03.217998 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:03.296396 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:03.296360 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:03.393717 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:03.393597 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:03.490654 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:03.490618 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" event={"ID":"1b4458c2-e0f3-4be4-9691-58e3f4f42d38","Type":"ContainerStarted","Data":"5139aee9bfa74a70df07a4092176b275c8ace7b4fa898ab3eada980ce8fcd221"} Jul 07 18:29:03.494554 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:03.494521 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:06.502883 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:06.502851 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" event={"ID":"1b4458c2-e0f3-4be4-9691-58e3f4f42d38","Type":"ContainerStarted","Data":"bcf9e97bc20495aac9953aed9871d9a2593a4a56e804d86fb21eae8cf64a9923"} Jul 07 18:29:07.507813 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:07.507777 2578 generic.go:356] "Generic (PLEG): container finished" podID="1b4458c2-e0f3-4be4-9691-58e3f4f42d38" containerID="bcf9e97bc20495aac9953aed9871d9a2593a4a56e804d86fb21eae8cf64a9923" exitCode=0 Jul 07 18:29:07.508277 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:07.507821 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" event={"ID":"1b4458c2-e0f3-4be4-9691-58e3f4f42d38","Type":"ContainerDied","Data":"bcf9e97bc20495aac9953aed9871d9a2593a4a56e804d86fb21eae8cf64a9923"} Jul 07 18:29:08.513805 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:08.513769 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" event={"ID":"1b4458c2-e0f3-4be4-9691-58e3f4f42d38","Type":"ContainerStarted","Data":"a9e55407c8c9b1187bf393459ac2b45cc077266a04ebacc016af70124e285ad8"} Jul 07 18:29:08.514279 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:08.513870 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" Jul 07 18:29:08.579203 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:08.579145 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" podStartSLOduration=2.346432715 podStartE2EDuration="6.579126421s" totalImagesPullingTime="3.150600624s" totalInitContainerRuntime="1.082093082s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:29:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:29:08.578610635 +0000 UTC m=+371.962404026" watchObservedRunningTime="2026-07-07 18:29:08.579126421 +0000 UTC m=+371.962919814" Jul 07 18:29:11.933029 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:11.932996 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:12.029222 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:12.029188 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:19.520034 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:19.520005 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7595c859c8-vk6nq" Jul 07 18:29:21.926692 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:21.926662 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:22.058235 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:22.058202 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:22.535572 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:22.535535 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-487nt"] Jul 07 18:29:22.538599 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:22.538579 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-487nt" Jul 07 18:29:22.542221 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:22.542191 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-m6crd\"" Jul 07 18:29:22.542349 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:22.542258 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 07 18:29:22.553144 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:22.553115 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-487nt"] Jul 07 18:29:22.582130 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:22.582100 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:22.667099 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:22.667054 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:22.722984 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:22.722944 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvdrq\" (UniqueName: \"kubernetes.io/projected/d5c83c3f-318d-4c87-ace1-01608156d78b-kube-api-access-pvdrq\") pod \"odh-model-controller-7998c67dc4-487nt\" (UID: \"d5c83c3f-318d-4c87-ace1-01608156d78b\") " pod="opendatahub/odh-model-controller-7998c67dc4-487nt" Jul 07 18:29:22.723162 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:22.723043 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5c83c3f-318d-4c87-ace1-01608156d78b-cert\") pod \"odh-model-controller-7998c67dc4-487nt\" (UID: \"d5c83c3f-318d-4c87-ace1-01608156d78b\") " pod="opendatahub/odh-model-controller-7998c67dc4-487nt" Jul 07 18:29:22.755987 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:22.755952 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:22.824190 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:22.824084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5c83c3f-318d-4c87-ace1-01608156d78b-cert\") pod \"odh-model-controller-7998c67dc4-487nt\" (UID: \"d5c83c3f-318d-4c87-ace1-01608156d78b\") " pod="opendatahub/odh-model-controller-7998c67dc4-487nt" Jul 07 18:29:22.824190 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:22.824128 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvdrq\" (UniqueName: \"kubernetes.io/projected/d5c83c3f-318d-4c87-ace1-01608156d78b-kube-api-access-pvdrq\") pod \"odh-model-controller-7998c67dc4-487nt\" (UID: \"d5c83c3f-318d-4c87-ace1-01608156d78b\") " pod="opendatahub/odh-model-controller-7998c67dc4-487nt" Jul 07 18:29:22.824423 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:29:22.824244 2578 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 07 18:29:22.824423 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:29:22.824341 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5c83c3f-318d-4c87-ace1-01608156d78b-cert podName:d5c83c3f-318d-4c87-ace1-01608156d78b nodeName:}" failed. No retries permitted until 2026-07-07 18:29:23.324298402 +0000 UTC m=+386.708091779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d5c83c3f-318d-4c87-ace1-01608156d78b-cert") pod "odh-model-controller-7998c67dc4-487nt" (UID: "d5c83c3f-318d-4c87-ace1-01608156d78b") : secret "odh-model-controller-webhook-cert" not found Jul 07 18:29:22.840734 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:22.840706 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvdrq\" (UniqueName: \"kubernetes.io/projected/d5c83c3f-318d-4c87-ace1-01608156d78b-kube-api-access-pvdrq\") pod \"odh-model-controller-7998c67dc4-487nt\" (UID: \"d5c83c3f-318d-4c87-ace1-01608156d78b\") " pod="opendatahub/odh-model-controller-7998c67dc4-487nt" Jul 07 18:29:23.053032 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:23.053000 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:23.210450 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:23.210365 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:23.268106 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:23.268076 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-llh67"] Jul 07 18:29:23.271440 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:23.271414 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-llh67" Jul 07 18:29:23.273791 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:23.273762 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-x4zkm\"" Jul 07 18:29:23.273910 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:23.273808 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 07 18:29:23.285094 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:23.285061 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-llh67"] Jul 07 18:29:23.328723 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:23.328688 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5c83c3f-318d-4c87-ace1-01608156d78b-cert\") pod \"odh-model-controller-7998c67dc4-487nt\" (UID: \"d5c83c3f-318d-4c87-ace1-01608156d78b\") " pod="opendatahub/odh-model-controller-7998c67dc4-487nt" Jul 07 18:29:23.329836 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:23.329810 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:23.331354 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:23.331312 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5c83c3f-318d-4c87-ace1-01608156d78b-cert\") pod \"odh-model-controller-7998c67dc4-487nt\" (UID: \"d5c83c3f-318d-4c87-ace1-01608156d78b\") " pod="opendatahub/odh-model-controller-7998c67dc4-487nt" Jul 07 18:29:23.429520 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:23.429478 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzrr7\" (UniqueName: \"kubernetes.io/projected/2f3548c4-1727-409f-88f2-2301a2ad2204-kube-api-access-jzrr7\") pod \"model-serving-api-54d89bd5fc-llh67\" (UID: \"2f3548c4-1727-409f-88f2-2301a2ad2204\") " pod="opendatahub/model-serving-api-54d89bd5fc-llh67" Jul 07 18:29:23.429520 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:23.429523 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f3548c4-1727-409f-88f2-2301a2ad2204-tls-certs\") pod \"model-serving-api-54d89bd5fc-llh67\" (UID: \"2f3548c4-1727-409f-88f2-2301a2ad2204\") " pod="opendatahub/model-serving-api-54d89bd5fc-llh67" Jul 07 18:29:23.448617 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:23.448585 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-487nt" Jul 07 18:29:23.530725 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:23.530679 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzrr7\" (UniqueName: \"kubernetes.io/projected/2f3548c4-1727-409f-88f2-2301a2ad2204-kube-api-access-jzrr7\") pod \"model-serving-api-54d89bd5fc-llh67\" (UID: \"2f3548c4-1727-409f-88f2-2301a2ad2204\") " pod="opendatahub/model-serving-api-54d89bd5fc-llh67" Jul 07 18:29:23.530906 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:23.530741 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f3548c4-1727-409f-88f2-2301a2ad2204-tls-certs\") pod \"model-serving-api-54d89bd5fc-llh67\" (UID: \"2f3548c4-1727-409f-88f2-2301a2ad2204\") " pod="opendatahub/model-serving-api-54d89bd5fc-llh67" Jul 07 18:29:23.533401 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:23.533343 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f3548c4-1727-409f-88f2-2301a2ad2204-tls-certs\") pod \"model-serving-api-54d89bd5fc-llh67\" (UID: \"2f3548c4-1727-409f-88f2-2301a2ad2204\") " pod="opendatahub/model-serving-api-54d89bd5fc-llh67" Jul 07 18:29:23.540502 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:23.540457 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzrr7\" (UniqueName: \"kubernetes.io/projected/2f3548c4-1727-409f-88f2-2301a2ad2204-kube-api-access-jzrr7\") pod \"model-serving-api-54d89bd5fc-llh67\" (UID: \"2f3548c4-1727-409f-88f2-2301a2ad2204\") " pod="opendatahub/model-serving-api-54d89bd5fc-llh67" Jul 07 18:29:23.583408 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:23.583376 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-llh67" Jul 07 18:29:23.596835 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:23.596757 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-487nt"] Jul 07 18:29:23.600344 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:29:23.600290 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c83c3f_318d_4c87_ace1_01608156d78b.slice/crio-66adbdaea5761c9f7ad6fa8292b00391f79add01d68d114048e73be040b376e5 WatchSource:0}: Error finding container 66adbdaea5761c9f7ad6fa8292b00391f79add01d68d114048e73be040b376e5: Status 404 returned error can't find the container with id 66adbdaea5761c9f7ad6fa8292b00391f79add01d68d114048e73be040b376e5 Jul 07 18:29:23.720917 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:23.720881 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-llh67"] Jul 07 18:29:23.724476 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:29:23.724446 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3548c4_1727_409f_88f2_2301a2ad2204.slice/crio-7492cb6ac9444bf1a9090fab9d37d4cc3a15a6780adaa4651db3bf3741f132e9 WatchSource:0}: Error finding container 7492cb6ac9444bf1a9090fab9d37d4cc3a15a6780adaa4651db3bf3741f132e9: Status 404 returned error can't find the container with id 7492cb6ac9444bf1a9090fab9d37d4cc3a15a6780adaa4651db3bf3741f132e9 Jul 07 18:29:24.571036 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:24.571002 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-llh67" event={"ID":"2f3548c4-1727-409f-88f2-2301a2ad2204","Type":"ContainerStarted","Data":"7492cb6ac9444bf1a9090fab9d37d4cc3a15a6780adaa4651db3bf3741f132e9"} Jul 07 18:29:24.573005 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:24.572929 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-487nt" event={"ID":"d5c83c3f-318d-4c87-ace1-01608156d78b","Type":"ContainerStarted","Data":"66adbdaea5761c9f7ad6fa8292b00391f79add01d68d114048e73be040b376e5"} Jul 07 18:29:25.447769 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:25.447736 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:25.536004 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:25.535969 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-f5865f688-kgtgc"] Jul 07 18:29:25.540731 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:25.540700 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f5865f688-kgtgc" Jul 07 18:29:25.543433 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:25.543407 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7srkw\"" Jul 07 18:29:25.543578 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:25.543450 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 18:29:25.543578 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:25.543407 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 18:29:25.543718 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:25.543701 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 18:29:25.544265 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:25.544246 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 18:29:25.547636 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:25.547616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107-tls-certs\") pod \"kube-auth-proxy-f5865f688-kgtgc\" (UID: \"73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107\") " pod="openshift-ingress/kube-auth-proxy-f5865f688-kgtgc" Jul 07 18:29:25.547758 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:25.547653 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107-tmp\") pod \"kube-auth-proxy-f5865f688-kgtgc\" (UID: \"73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107\") " pod="openshift-ingress/kube-auth-proxy-f5865f688-kgtgc" Jul 07 18:29:25.548115 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:25.547765 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtp2z\" (UniqueName: \"kubernetes.io/projected/73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107-kube-api-access-wtp2z\") pod \"kube-auth-proxy-f5865f688-kgtgc\" (UID: \"73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107\") " pod="openshift-ingress/kube-auth-proxy-f5865f688-kgtgc" Jul 07 18:29:25.549752 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:25.549689 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f5865f688-kgtgc"] Jul 07 18:29:25.611294 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:25.611247 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:25.649303 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:25.649268 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtp2z\" (UniqueName: \"kubernetes.io/projected/73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107-kube-api-access-wtp2z\") pod \"kube-auth-proxy-f5865f688-kgtgc\" (UID: \"73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107\") " pod="openshift-ingress/kube-auth-proxy-f5865f688-kgtgc" Jul 07 18:29:25.649494 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:25.649312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107-tls-certs\") pod \"kube-auth-proxy-f5865f688-kgtgc\" (UID: \"73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107\") " pod="openshift-ingress/kube-auth-proxy-f5865f688-kgtgc" Jul 07 18:29:25.649494 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:25.649362 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107-tmp\") pod \"kube-auth-proxy-f5865f688-kgtgc\" (UID: \"73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107\") " pod="openshift-ingress/kube-auth-proxy-f5865f688-kgtgc" Jul 07 18:29:25.651960 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:25.651937 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107-tmp\") pod \"kube-auth-proxy-f5865f688-kgtgc\" (UID: \"73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107\") " pod="openshift-ingress/kube-auth-proxy-f5865f688-kgtgc" Jul 07 18:29:25.652129 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:25.652107 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107-tls-certs\") pod \"kube-auth-proxy-f5865f688-kgtgc\" (UID: \"73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107\") " pod="openshift-ingress/kube-auth-proxy-f5865f688-kgtgc" Jul 07 18:29:25.657891 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:25.657861 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtp2z\" (UniqueName: \"kubernetes.io/projected/73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107-kube-api-access-wtp2z\") pod \"kube-auth-proxy-f5865f688-kgtgc\" (UID: \"73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107\") " pod="openshift-ingress/kube-auth-proxy-f5865f688-kgtgc" Jul 07 18:29:25.854670 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:25.854632 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f5865f688-kgtgc" Jul 07 18:29:26.956240 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:26.956210 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f5865f688-kgtgc"] Jul 07 18:29:26.958739 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:29:26.958701 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c3ff7b_0cc1_4a70_bb0b_16d0c8b90107.slice/crio-671170c3ffe80b66be2c1b500bca4c8c138711a2474df349f6e8f9821aebb053 WatchSource:0}: Error finding container 671170c3ffe80b66be2c1b500bca4c8c138711a2474df349f6e8f9821aebb053: Status 404 returned error can't find the container with id 671170c3ffe80b66be2c1b500bca4c8c138711a2474df349f6e8f9821aebb053 Jul 07 18:29:27.061822 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.061718 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:27.064064 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.064043 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj"] Jul 07 18:29:27.067421 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.067406 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.071406 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.071381 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 07 18:29:27.071780 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.071764 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 07 18:29:27.071861 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.071826 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-m7jp2\"" Jul 07 18:29:27.071917 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.071867 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 18:29:27.073660 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.073637 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 07 18:29:27.087384 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.087351 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj"] Jul 07 18:29:27.153186 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.153148 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:27.160205 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.160170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-token\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.160406 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.160236 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e49d32-c370-4c2d-9f4d-11a0e03d8932-local-certs\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.160406 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.160261 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpgjb\" (UniqueName: \"kubernetes.io/projected/30e49d32-c370-4c2d-9f4d-11a0e03d8932-kube-api-access-gpgjb\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.160406 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.160352 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30e49d32-c370-4c2d-9f4d-11a0e03d8932-cacerts\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.160406 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.160403 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.160576 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.160453 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.160576 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.160479 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.261750 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.261710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e49d32-c370-4c2d-9f4d-11a0e03d8932-local-certs\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.261750 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.261748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpgjb\" (UniqueName: \"kubernetes.io/projected/30e49d32-c370-4c2d-9f4d-11a0e03d8932-kube-api-access-gpgjb\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.261987 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.261772 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30e49d32-c370-4c2d-9f4d-11a0e03d8932-cacerts\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.261987 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.261806 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.261987 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.261849 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.261987 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.261876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.261987 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.261921 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-token\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.262556 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.262529 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.264120 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.264090 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e49d32-c370-4c2d-9f4d-11a0e03d8932-local-certs\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.264201 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.264124 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.264301 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.264281 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30e49d32-c370-4c2d-9f4d-11a0e03d8932-cacerts\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.264365 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.264296 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.273610 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.273577 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-token\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.273737 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.273723 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpgjb\" (UniqueName: \"kubernetes.io/projected/30e49d32-c370-4c2d-9f4d-11a0e03d8932-kube-api-access-gpgjb\") pod \"istiod-openshift-gateway-7574459656-r2lbj\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.377569 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.377468 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:27.528827 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.528795 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj"] Jul 07 18:29:27.532000 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:29:27.531969 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e49d32_c370_4c2d_9f4d_11a0e03d8932.slice/crio-3a8fbcf7fda0acbe52083b8c1fc0d8119518885ad570dfdbda6881d190b2f4bc WatchSource:0}: Error finding container 3a8fbcf7fda0acbe52083b8c1fc0d8119518885ad570dfdbda6881d190b2f4bc: Status 404 returned error can't find the container with id 3a8fbcf7fda0acbe52083b8c1fc0d8119518885ad570dfdbda6881d190b2f4bc Jul 07 18:29:27.590191 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.590152 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-487nt" event={"ID":"d5c83c3f-318d-4c87-ace1-01608156d78b","Type":"ContainerStarted","Data":"836732e4ba7fa10740f695c64e39d5fe1a7c13820453ec6792622113bcfd8130"} Jul 07 18:29:27.590402 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.590308 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-487nt" Jul 07 18:29:27.592105 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.592051 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-llh67" event={"ID":"2f3548c4-1727-409f-88f2-2301a2ad2204","Type":"ContainerStarted","Data":"d46b3318ba590a4d279bda20aa69ffd26d9294357cabd8ec23c2682056585a11"} Jul 07 18:29:27.592269 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.592251 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-llh67" Jul 07 18:29:27.593647 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.593603 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" event={"ID":"30e49d32-c370-4c2d-9f4d-11a0e03d8932","Type":"ContainerStarted","Data":"3a8fbcf7fda0acbe52083b8c1fc0d8119518885ad570dfdbda6881d190b2f4bc"} Jul 07 18:29:27.595410 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.595369 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f5865f688-kgtgc" event={"ID":"73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107","Type":"ContainerStarted","Data":"671170c3ffe80b66be2c1b500bca4c8c138711a2474df349f6e8f9821aebb053"} Jul 07 18:29:27.610786 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.610701 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-487nt" podStartSLOduration=2.3704569380000002 podStartE2EDuration="5.610679067s" totalImagesPullingTime="3.240222129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:29:27.608858725 +0000 UTC m=+390.992652118" watchObservedRunningTime="2026-07-07 18:29:27.610679067 +0000 UTC m=+390.994472463" Jul 07 18:29:27.630868 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:27.630768 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-llh67" podStartSLOduration=1.567641874 podStartE2EDuration="4.630750722s" totalImagesPullingTime="3.063108848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:29:27.630246638 +0000 UTC m=+391.014040033" watchObservedRunningTime="2026-07-07 18:29:27.630750722 +0000 UTC m=+391.014544116" Jul 07 18:29:28.171818 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:28.171720 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:28.382644 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:28.382603 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:30.349480 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:30.349429 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:29:30.349748 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:30.349508 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:29:30.611113 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:30.611009 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" event={"ID":"30e49d32-c370-4c2d-9f4d-11a0e03d8932","Type":"ContainerStarted","Data":"762e99c6ae400eb0618a19252be1f94e7a56db850db79a1a149c0d3fb3b1669f"} Jul 07 18:29:30.611372 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:30.611354 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:30.612749 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:30.612722 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f5865f688-kgtgc" event={"ID":"73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107","Type":"ContainerStarted","Data":"1b51f1c79f8c2cf245dac41ac626b5c489bfb60c1a2f2ed52094498fe55d1be2"} Jul 07 18:29:30.657262 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:30.657200 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" podStartSLOduration=0.842257152 podStartE2EDuration="3.657184118s" totalImagesPullingTime="2.814926966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:29:30.638465668 +0000 UTC m=+394.022259064" watchObservedRunningTime="2026-07-07 18:29:30.657184118 +0000 UTC m=+394.040977512" Jul 07 18:29:30.658347 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:30.658289 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-f5865f688-kgtgc" podStartSLOduration=2.271941514 podStartE2EDuration="5.658277113s" totalImagesPullingTime="3.386335599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:29:30.656879146 +0000 UTC m=+394.040672554" watchObservedRunningTime="2026-07-07 18:29:30.658277113 +0000 UTC m=+394.042070506" Jul 07 18:29:30.877285 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:30.877183 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:30.987922 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:30.987883 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:31.407152 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:31.407115 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:31.560244 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:31.560211 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:31.618041 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:31.618017 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:31.803273 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:31.803242 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:35.571186 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.571153 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:35.673209 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.673179 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-hkzl2"] Jul 07 18:29:35.681124 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.681098 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hkzl2" Jul 07 18:29:35.681353 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.681314 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-v2lrb"] Jul 07 18:29:35.683679 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.683658 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-gb6qt\"" Jul 07 18:29:35.683773 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.683736 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 07 18:29:35.684771 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.684754 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v2lrb" Jul 07 18:29:35.687806 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.687788 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-l5cjf\"" Jul 07 18:29:35.687872 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.687829 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 07 18:29:35.695084 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.695063 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-hkzl2"] Jul 07 18:29:35.699184 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.699163 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-v2lrb"] Jul 07 18:29:35.728783 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.728747 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms2j9\" (UniqueName: \"kubernetes.io/projected/018abded-bab5-4066-803b-f387501cc678-kube-api-access-ms2j9\") pod \"kserve-controller-manager-d6f4b65b8-hkzl2\" (UID: \"018abded-bab5-4066-803b-f387501cc678\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hkzl2" Jul 07 18:29:35.728947 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.728788 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa343ac0-bade-49dd-9fc5-35778b5e2ee7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-v2lrb\" (UID: \"fa343ac0-bade-49dd-9fc5-35778b5e2ee7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v2lrb" Jul 07 18:29:35.728947 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.728849 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-454kj\" (UniqueName: \"kubernetes.io/projected/fa343ac0-bade-49dd-9fc5-35778b5e2ee7-kube-api-access-454kj\") pod \"llmisvc-controller-manager-777c6bd7d5-v2lrb\" (UID: \"fa343ac0-bade-49dd-9fc5-35778b5e2ee7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v2lrb" Jul 07 18:29:35.728947 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.728886 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/018abded-bab5-4066-803b-f387501cc678-cert\") pod \"kserve-controller-manager-d6f4b65b8-hkzl2\" (UID: \"018abded-bab5-4066-803b-f387501cc678\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hkzl2" Jul 07 18:29:35.743905 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.743876 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:35.822933 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.822854 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:35.830022 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.830002 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa343ac0-bade-49dd-9fc5-35778b5e2ee7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-v2lrb\" (UID: \"fa343ac0-bade-49dd-9fc5-35778b5e2ee7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v2lrb" Jul 07 18:29:35.830086 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.830040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-454kj\" (UniqueName: \"kubernetes.io/projected/fa343ac0-bade-49dd-9fc5-35778b5e2ee7-kube-api-access-454kj\") pod \"llmisvc-controller-manager-777c6bd7d5-v2lrb\" (UID: \"fa343ac0-bade-49dd-9fc5-35778b5e2ee7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v2lrb" Jul 07 18:29:35.830086 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.830058 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/018abded-bab5-4066-803b-f387501cc678-cert\") pod \"kserve-controller-manager-d6f4b65b8-hkzl2\" (UID: \"018abded-bab5-4066-803b-f387501cc678\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hkzl2" Jul 07 18:29:35.830192 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.830143 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms2j9\" (UniqueName: \"kubernetes.io/projected/018abded-bab5-4066-803b-f387501cc678-kube-api-access-ms2j9\") pod \"kserve-controller-manager-d6f4b65b8-hkzl2\" (UID: \"018abded-bab5-4066-803b-f387501cc678\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hkzl2" Jul 07 18:29:35.830192 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:29:35.830151 2578 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 07 18:29:35.830290 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:29:35.830219 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa343ac0-bade-49dd-9fc5-35778b5e2ee7-cert podName:fa343ac0-bade-49dd-9fc5-35778b5e2ee7 nodeName:}" failed. No retries permitted until 2026-07-07 18:29:36.330197742 +0000 UTC m=+399.713991116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fa343ac0-bade-49dd-9fc5-35778b5e2ee7-cert") pod "llmisvc-controller-manager-777c6bd7d5-v2lrb" (UID: "fa343ac0-bade-49dd-9fc5-35778b5e2ee7") : secret "llmisvc-webhook-server-cert" not found Jul 07 18:29:35.832610 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.832588 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/018abded-bab5-4066-803b-f387501cc678-cert\") pod \"kserve-controller-manager-d6f4b65b8-hkzl2\" (UID: \"018abded-bab5-4066-803b-f387501cc678\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hkzl2" Jul 07 18:29:35.839964 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.839937 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-454kj\" (UniqueName: \"kubernetes.io/projected/fa343ac0-bade-49dd-9fc5-35778b5e2ee7-kube-api-access-454kj\") pod \"llmisvc-controller-manager-777c6bd7d5-v2lrb\" (UID: \"fa343ac0-bade-49dd-9fc5-35778b5e2ee7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v2lrb" Jul 07 18:29:35.840128 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.840111 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms2j9\" (UniqueName: \"kubernetes.io/projected/018abded-bab5-4066-803b-f387501cc678-kube-api-access-ms2j9\") pod \"kserve-controller-manager-d6f4b65b8-hkzl2\" (UID: \"018abded-bab5-4066-803b-f387501cc678\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hkzl2" Jul 07 18:29:35.992980 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:35.992947 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hkzl2" Jul 07 18:29:36.157348 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.157305 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-hkzl2"] Jul 07 18:29:36.158926 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:29:36.158901 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018abded_bab5_4066_803b_f387501cc678.slice/crio-9cf861aa3da0c890a2786d297d978ce2111fb8b4e5a0c94f16e60aade1951d9b WatchSource:0}: Error finding container 9cf861aa3da0c890a2786d297d978ce2111fb8b4e5a0c94f16e60aade1951d9b: Status 404 returned error can't find the container with id 9cf861aa3da0c890a2786d297d978ce2111fb8b4e5a0c94f16e60aade1951d9b Jul 07 18:29:36.325983 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.325949 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:36.334067 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.333998 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa343ac0-bade-49dd-9fc5-35778b5e2ee7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-v2lrb\" (UID: \"fa343ac0-bade-49dd-9fc5-35778b5e2ee7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v2lrb" Jul 07 18:29:36.336293 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.336266 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa343ac0-bade-49dd-9fc5-35778b5e2ee7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-v2lrb\" (UID: \"fa343ac0-bade-49dd-9fc5-35778b5e2ee7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v2lrb" Jul 07 18:29:36.344473 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.344448 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj"] Jul 07 18:29:36.344666 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.344642 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" podUID="30e49d32-c370-4c2d-9f4d-11a0e03d8932" containerName="discovery" containerID="cri-o://762e99c6ae400eb0618a19252be1f94e7a56db850db79a1a149c0d3fb3b1669f" gracePeriod=30 Jul 07 18:29:36.576080 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.576057 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:36.599023 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.598931 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v2lrb" Jul 07 18:29:36.635402 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.635367 2578 generic.go:356] "Generic (PLEG): container finished" podID="30e49d32-c370-4c2d-9f4d-11a0e03d8932" containerID="762e99c6ae400eb0618a19252be1f94e7a56db850db79a1a149c0d3fb3b1669f" exitCode=0 Jul 07 18:29:36.635572 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.635414 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" Jul 07 18:29:36.635572 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.635450 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" event={"ID":"30e49d32-c370-4c2d-9f4d-11a0e03d8932","Type":"ContainerDied","Data":"762e99c6ae400eb0618a19252be1f94e7a56db850db79a1a149c0d3fb3b1669f"} Jul 07 18:29:36.635572 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.635495 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj" event={"ID":"30e49d32-c370-4c2d-9f4d-11a0e03d8932","Type":"ContainerDied","Data":"3a8fbcf7fda0acbe52083b8c1fc0d8119518885ad570dfdbda6881d190b2f4bc"} Jul 07 18:29:36.635572 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.635522 2578 scope.go:117] "RemoveContainer" containerID="762e99c6ae400eb0618a19252be1f94e7a56db850db79a1a149c0d3fb3b1669f" Jul 07 18:29:36.636441 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.636406 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-kubeconfig\") pod \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " Jul 07 18:29:36.636441 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.636440 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-csr-ca-configmap\") pod \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " Jul 07 18:29:36.636600 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.636482 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30e49d32-c370-4c2d-9f4d-11a0e03d8932-cacerts\") pod \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " Jul 07 18:29:36.636600 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.636524 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpgjb\" (UniqueName: \"kubernetes.io/projected/30e49d32-c370-4c2d-9f4d-11a0e03d8932-kube-api-access-gpgjb\") pod \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " Jul 07 18:29:36.636600 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.636556 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e49d32-c370-4c2d-9f4d-11a0e03d8932-local-certs\") pod \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " Jul 07 18:29:36.636600 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.636581 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-csr-dns-cert\") pod \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " Jul 07 18:29:36.636862 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.636613 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-token\") pod \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\" (UID: \"30e49d32-c370-4c2d-9f4d-11a0e03d8932\") " Jul 07 18:29:36.636862 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.636660 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hkzl2" event={"ID":"018abded-bab5-4066-803b-f387501cc678","Type":"ContainerStarted","Data":"9cf861aa3da0c890a2786d297d978ce2111fb8b4e5a0c94f16e60aade1951d9b"} Jul 07 18:29:36.637959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.637617 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "30e49d32-c370-4c2d-9f4d-11a0e03d8932" (UID: "30e49d32-c370-4c2d-9f4d-11a0e03d8932"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:29:36.639250 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.639210 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "30e49d32-c370-4c2d-9f4d-11a0e03d8932" (UID: "30e49d32-c370-4c2d-9f4d-11a0e03d8932"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:29:36.639362 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.639244 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-token" (OuterVolumeSpecName: "istio-token") pod "30e49d32-c370-4c2d-9f4d-11a0e03d8932" (UID: "30e49d32-c370-4c2d-9f4d-11a0e03d8932"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:29:36.639814 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.639786 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e49d32-c370-4c2d-9f4d-11a0e03d8932-kube-api-access-gpgjb" (OuterVolumeSpecName: "kube-api-access-gpgjb") pod "30e49d32-c370-4c2d-9f4d-11a0e03d8932" (UID: "30e49d32-c370-4c2d-9f4d-11a0e03d8932"). InnerVolumeSpecName "kube-api-access-gpgjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:29:36.641175 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.641152 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e49d32-c370-4c2d-9f4d-11a0e03d8932-local-certs" (OuterVolumeSpecName: "local-certs") pod "30e49d32-c370-4c2d-9f4d-11a0e03d8932" (UID: "30e49d32-c370-4c2d-9f4d-11a0e03d8932"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:36.641266 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.641242 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30e49d32-c370-4c2d-9f4d-11a0e03d8932-cacerts" (OuterVolumeSpecName: "cacerts") pod "30e49d32-c370-4c2d-9f4d-11a0e03d8932" (UID: "30e49d32-c370-4c2d-9f4d-11a0e03d8932"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:29:36.641370 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.641342 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "30e49d32-c370-4c2d-9f4d-11a0e03d8932" (UID: "30e49d32-c370-4c2d-9f4d-11a0e03d8932"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:29:36.645160 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.645142 2578 scope.go:117] "RemoveContainer" containerID="762e99c6ae400eb0618a19252be1f94e7a56db850db79a1a149c0d3fb3b1669f" Jul 07 18:29:36.645427 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:29:36.645407 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"762e99c6ae400eb0618a19252be1f94e7a56db850db79a1a149c0d3fb3b1669f\": container with ID starting with 762e99c6ae400eb0618a19252be1f94e7a56db850db79a1a149c0d3fb3b1669f not found: ID does not exist" containerID="762e99c6ae400eb0618a19252be1f94e7a56db850db79a1a149c0d3fb3b1669f" Jul 07 18:29:36.645490 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.645440 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762e99c6ae400eb0618a19252be1f94e7a56db850db79a1a149c0d3fb3b1669f"} err="failed to get container status \"762e99c6ae400eb0618a19252be1f94e7a56db850db79a1a149c0d3fb3b1669f\": rpc error: code = NotFound desc = could not find container \"762e99c6ae400eb0618a19252be1f94e7a56db850db79a1a149c0d3fb3b1669f\": container with ID starting with 762e99c6ae400eb0618a19252be1f94e7a56db850db79a1a149c0d3fb3b1669f not found: ID does not exist" Jul 07 18:29:36.737677 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.737641 2578 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-kubeconfig\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:29:36.737677 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.737669 2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-csr-ca-configmap\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:29:36.737677 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.737679 2578 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30e49d32-c370-4c2d-9f4d-11a0e03d8932-cacerts\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:29:36.737677 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.737687 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpgjb\" (UniqueName: \"kubernetes.io/projected/30e49d32-c370-4c2d-9f4d-11a0e03d8932-kube-api-access-gpgjb\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:29:36.738017 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.737697 2578 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e49d32-c370-4c2d-9f4d-11a0e03d8932-local-certs\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:29:36.738017 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.737705 2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-csr-dns-cert\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:29:36.738017 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.737713 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e49d32-c370-4c2d-9f4d-11a0e03d8932-istio-token\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:29:36.767002 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.766982 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-v2lrb"] Jul 07 18:29:36.769027 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:29:36.769005 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfa343ac0_bade_49dd_9fc5_35778b5e2ee7.slice/crio-ca0adc41fcaaa5d2b42b9b13ada67d32d232b4ffcf54c96f24d38163ef8e657d WatchSource:0}: Error finding container ca0adc41fcaaa5d2b42b9b13ada67d32d232b4ffcf54c96f24d38163ef8e657d: Status 404 returned error can't find the container with id ca0adc41fcaaa5d2b42b9b13ada67d32d232b4ffcf54c96f24d38163ef8e657d Jul 07 18:29:36.971376 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.971345 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj"] Jul 07 18:29:36.983563 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.983540 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r2lbj"] Jul 07 18:29:36.984831 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:36.984807 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:37.122675 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:37.122641 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:37.196374 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:37.196300 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30e49d32-c370-4c2d-9f4d-11a0e03d8932" path="/var/lib/kubelet/pods/30e49d32-c370-4c2d-9f4d-11a0e03d8932/volumes" Jul 07 18:29:37.228356 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:37.228310 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:37.643492 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:37.643456 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v2lrb" event={"ID":"fa343ac0-bade-49dd-9fc5-35778b5e2ee7","Type":"ContainerStarted","Data":"ca0adc41fcaaa5d2b42b9b13ada67d32d232b4ffcf54c96f24d38163ef8e657d"} Jul 07 18:29:38.604218 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:38.604170 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-487nt" Jul 07 18:29:38.606825 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:38.606795 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-llh67" Jul 07 18:29:39.654695 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:39.654655 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hkzl2" event={"ID":"018abded-bab5-4066-803b-f387501cc678","Type":"ContainerStarted","Data":"60641bb3888e04ab557518a4fc71581f1749e86bcb60cc098a47133372abc20a"} Jul 07 18:29:39.655167 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:39.654773 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hkzl2" Jul 07 18:29:39.684116 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:39.684056 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hkzl2" podStartSLOduration=1.759641459 podStartE2EDuration="4.684033631s" totalImagesPullingTime="2.924392172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:29:39.682553736 +0000 UTC m=+403.066347157" watchObservedRunningTime="2026-07-07 18:29:39.684033631 +0000 UTC m=+403.067827026" Jul 07 18:29:40.659907 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:40.659873 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v2lrb" event={"ID":"fa343ac0-bade-49dd-9fc5-35778b5e2ee7","Type":"ContainerStarted","Data":"61af20184abc8982bb684741924f04a3cc4383ba3fef9a4666f5e632b7eadde6"} Jul 07 18:29:40.660315 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:40.660175 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v2lrb" Jul 07 18:29:40.703476 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:29:40.703425 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v2lrb" podStartSLOduration=2.436302431 podStartE2EDuration="5.703407927s" totalImagesPullingTime="3.267105496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:29:40.69958354 +0000 UTC m=+404.083376933" watchObservedRunningTime="2026-07-07 18:29:40.703407927 +0000 UTC m=+404.087201321" Jul 07 18:30:10.664985 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:10.664957 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hkzl2" Jul 07 18:30:12.670480 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:12.670449 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v2lrb" Jul 07 18:30:23.507300 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:23.507214 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:23.598414 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:23.598370 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:23.636508 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:23.636485 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lwc4j"] Jul 07 18:30:23.636745 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:23.636733 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30e49d32-c370-4c2d-9f4d-11a0e03d8932" containerName="discovery" Jul 07 18:30:23.636787 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:23.636746 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="30e49d32-c370-4c2d-9f4d-11a0e03d8932" containerName="discovery" Jul 07 18:30:23.636826 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:23.636795 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="30e49d32-c370-4c2d-9f4d-11a0e03d8932" containerName="discovery" Jul 07 18:30:23.638706 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:23.638686 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lwc4j" Jul 07 18:30:23.641201 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:23.641180 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 18:30:23.641307 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:23.641213 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 18:30:23.642038 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:23.642018 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-g5qfm\"" Jul 07 18:30:23.660503 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:23.660478 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lwc4j"] Jul 07 18:30:23.703443 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:23.703417 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:23.804761 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:23.804710 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k86ds\" (UniqueName: \"kubernetes.io/projected/3ebdd345-9423-4be2-8003-ac98c2c8d4c8-kube-api-access-k86ds\") pod \"kuadrant-operator-catalog-lwc4j\" (UID: \"3ebdd345-9423-4be2-8003-ac98c2c8d4c8\") " pod="kuadrant-system/kuadrant-operator-catalog-lwc4j" Jul 07 18:30:23.905144 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:23.905121 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k86ds\" (UniqueName: \"kubernetes.io/projected/3ebdd345-9423-4be2-8003-ac98c2c8d4c8-kube-api-access-k86ds\") pod \"kuadrant-operator-catalog-lwc4j\" (UID: \"3ebdd345-9423-4be2-8003-ac98c2c8d4c8\") " pod="kuadrant-system/kuadrant-operator-catalog-lwc4j" Jul 07 18:30:23.919670 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:23.919653 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k86ds\" (UniqueName: \"kubernetes.io/projected/3ebdd345-9423-4be2-8003-ac98c2c8d4c8-kube-api-access-k86ds\") pod \"kuadrant-operator-catalog-lwc4j\" (UID: \"3ebdd345-9423-4be2-8003-ac98c2c8d4c8\") " pod="kuadrant-system/kuadrant-operator-catalog-lwc4j" Jul 07 18:30:23.948562 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:23.948539 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lwc4j" Jul 07 18:30:23.999775 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:23.999741 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lwc4j"] Jul 07 18:30:24.073579 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:24.073488 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lwc4j"] Jul 07 18:30:24.076681 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:30:24.076653 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ebdd345_9423_4be2_8003_ac98c2c8d4c8.slice/crio-fd7cffac9db6804ab834fb1b1455ef2cf25551ab751a4159331994e75d95bd0c WatchSource:0}: Error finding container fd7cffac9db6804ab834fb1b1455ef2cf25551ab751a4159331994e75d95bd0c: Status 404 returned error can't find the container with id fd7cffac9db6804ab834fb1b1455ef2cf25551ab751a4159331994e75d95bd0c Jul 07 18:30:24.208188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:24.208151 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mjf79"] Jul 07 18:30:24.211414 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:24.211396 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mjf79" Jul 07 18:30:24.220653 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:24.220626 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mjf79"] Jul 07 18:30:24.308029 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:24.307996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ffh5\" (UniqueName: \"kubernetes.io/projected/4b300561-22d7-4581-a149-ee9847db2eb3-kube-api-access-5ffh5\") pod \"kuadrant-operator-catalog-mjf79\" (UID: \"4b300561-22d7-4581-a149-ee9847db2eb3\") " pod="kuadrant-system/kuadrant-operator-catalog-mjf79" Jul 07 18:30:24.408707 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:24.408624 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ffh5\" (UniqueName: \"kubernetes.io/projected/4b300561-22d7-4581-a149-ee9847db2eb3-kube-api-access-5ffh5\") pod \"kuadrant-operator-catalog-mjf79\" (UID: \"4b300561-22d7-4581-a149-ee9847db2eb3\") " pod="kuadrant-system/kuadrant-operator-catalog-mjf79" Jul 07 18:30:24.417826 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:24.417799 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ffh5\" (UniqueName: \"kubernetes.io/projected/4b300561-22d7-4581-a149-ee9847db2eb3-kube-api-access-5ffh5\") pod \"kuadrant-operator-catalog-mjf79\" (UID: \"4b300561-22d7-4581-a149-ee9847db2eb3\") " pod="kuadrant-system/kuadrant-operator-catalog-mjf79" Jul 07 18:30:24.437544 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:24.437515 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:24.521607 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:24.521568 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mjf79" Jul 07 18:30:24.643648 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:24.643616 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:24.647521 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:24.647498 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mjf79"] Jul 07 18:30:24.650167 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:30:24.650143 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b300561_22d7_4581_a149_ee9847db2eb3.slice/crio-0c9412a6c869f0e4c62e4a6634b104e9a7aa30c22b66a9365bd60d5d4ef2e0f4 WatchSource:0}: Error finding container 0c9412a6c869f0e4c62e4a6634b104e9a7aa30c22b66a9365bd60d5d4ef2e0f4: Status 404 returned error can't find the container with id 0c9412a6c869f0e4c62e4a6634b104e9a7aa30c22b66a9365bd60d5d4ef2e0f4 Jul 07 18:30:24.823248 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:24.823215 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mjf79" event={"ID":"4b300561-22d7-4581-a149-ee9847db2eb3","Type":"ContainerStarted","Data":"0c9412a6c869f0e4c62e4a6634b104e9a7aa30c22b66a9365bd60d5d4ef2e0f4"} Jul 07 18:30:24.824246 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:24.824225 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lwc4j" event={"ID":"3ebdd345-9423-4be2-8003-ac98c2c8d4c8","Type":"ContainerStarted","Data":"fd7cffac9db6804ab834fb1b1455ef2cf25551ab751a4159331994e75d95bd0c"} Jul 07 18:30:26.835121 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:26.835082 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mjf79" event={"ID":"4b300561-22d7-4581-a149-ee9847db2eb3","Type":"ContainerStarted","Data":"954676f3f6ec5448a0b37652766fca70c399df6e324ab74481dc2924219b2b67"} Jul 07 18:30:26.836670 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:26.836639 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lwc4j" event={"ID":"3ebdd345-9423-4be2-8003-ac98c2c8d4c8","Type":"ContainerStarted","Data":"9fcdfa252f40473675ae6eeba8a62c957edd26be79d48a975ce6fc2c4320af1d"} Jul 07 18:30:26.836807 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:26.836778 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-lwc4j" podUID="3ebdd345-9423-4be2-8003-ac98c2c8d4c8" containerName="registry-server" containerID="cri-o://9fcdfa252f40473675ae6eeba8a62c957edd26be79d48a975ce6fc2c4320af1d" gracePeriod=2 Jul 07 18:30:26.853550 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:26.853508 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-mjf79" podStartSLOduration=0.875400187 podStartE2EDuration="2.853491015s" totalImagesPullingTime="1.978090828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:30:26.853041244 +0000 UTC m=+450.236834662" watchObservedRunningTime="2026-07-07 18:30:26.853491015 +0000 UTC m=+450.237284409" Jul 07 18:30:26.878011 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:26.877905 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-lwc4j" podStartSLOduration=1.322608796 podStartE2EDuration="3.87788643s" totalImagesPullingTime="2.555277634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:30:26.877047749 +0000 UTC m=+450.260841142" watchObservedRunningTime="2026-07-07 18:30:26.87788643 +0000 UTC m=+450.261679841" Jul 07 18:30:27.082436 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:27.082408 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lwc4j" Jul 07 18:30:27.233044 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:27.233012 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k86ds\" (UniqueName: \"kubernetes.io/projected/3ebdd345-9423-4be2-8003-ac98c2c8d4c8-kube-api-access-k86ds\") pod \"3ebdd345-9423-4be2-8003-ac98c2c8d4c8\" (UID: \"3ebdd345-9423-4be2-8003-ac98c2c8d4c8\") " Jul 07 18:30:27.235236 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:27.235212 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ebdd345-9423-4be2-8003-ac98c2c8d4c8-kube-api-access-k86ds" (OuterVolumeSpecName: "kube-api-access-k86ds") pod "3ebdd345-9423-4be2-8003-ac98c2c8d4c8" (UID: "3ebdd345-9423-4be2-8003-ac98c2c8d4c8"). InnerVolumeSpecName "kube-api-access-k86ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:30:27.334309 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:27.334269 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k86ds\" (UniqueName: \"kubernetes.io/projected/3ebdd345-9423-4be2-8003-ac98c2c8d4c8-kube-api-access-k86ds\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:30:27.842205 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:27.842170 2578 generic.go:356] "Generic (PLEG): container finished" podID="3ebdd345-9423-4be2-8003-ac98c2c8d4c8" containerID="9fcdfa252f40473675ae6eeba8a62c957edd26be79d48a975ce6fc2c4320af1d" exitCode=0 Jul 07 18:30:27.842698 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:27.842220 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lwc4j" Jul 07 18:30:27.842698 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:27.842260 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lwc4j" event={"ID":"3ebdd345-9423-4be2-8003-ac98c2c8d4c8","Type":"ContainerDied","Data":"9fcdfa252f40473675ae6eeba8a62c957edd26be79d48a975ce6fc2c4320af1d"} Jul 07 18:30:27.842698 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:27.842304 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lwc4j" event={"ID":"3ebdd345-9423-4be2-8003-ac98c2c8d4c8","Type":"ContainerDied","Data":"fd7cffac9db6804ab834fb1b1455ef2cf25551ab751a4159331994e75d95bd0c"} Jul 07 18:30:27.842698 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:27.842338 2578 scope.go:117] "RemoveContainer" containerID="9fcdfa252f40473675ae6eeba8a62c957edd26be79d48a975ce6fc2c4320af1d" Jul 07 18:30:27.850582 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:27.850561 2578 scope.go:117] "RemoveContainer" containerID="9fcdfa252f40473675ae6eeba8a62c957edd26be79d48a975ce6fc2c4320af1d" Jul 07 18:30:27.850859 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:30:27.850838 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fcdfa252f40473675ae6eeba8a62c957edd26be79d48a975ce6fc2c4320af1d\": container with ID starting with 9fcdfa252f40473675ae6eeba8a62c957edd26be79d48a975ce6fc2c4320af1d not found: ID does not exist" containerID="9fcdfa252f40473675ae6eeba8a62c957edd26be79d48a975ce6fc2c4320af1d" Jul 07 18:30:27.850954 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:27.850865 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fcdfa252f40473675ae6eeba8a62c957edd26be79d48a975ce6fc2c4320af1d"} err="failed to get container status \"9fcdfa252f40473675ae6eeba8a62c957edd26be79d48a975ce6fc2c4320af1d\": rpc error: code = NotFound desc = could not find container \"9fcdfa252f40473675ae6eeba8a62c957edd26be79d48a975ce6fc2c4320af1d\": container with ID starting with 9fcdfa252f40473675ae6eeba8a62c957edd26be79d48a975ce6fc2c4320af1d not found: ID does not exist" Jul 07 18:30:27.864549 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:27.864506 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lwc4j"] Jul 07 18:30:27.868669 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:27.868638 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lwc4j"] Jul 07 18:30:29.197000 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:29.196969 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ebdd345-9423-4be2-8003-ac98c2c8d4c8" path="/var/lib/kubelet/pods/3ebdd345-9423-4be2-8003-ac98c2c8d4c8/volumes" Jul 07 18:30:34.522468 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:34.522428 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-mjf79" Jul 07 18:30:34.522904 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:34.522493 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-mjf79" Jul 07 18:30:34.546207 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:34.546184 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-mjf79" Jul 07 18:30:34.893214 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:34.893157 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-mjf79" Jul 07 18:30:35.040826 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:35.040795 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:38.248276 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.248240 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57f5bb7b7f-8mfgl"] Jul 07 18:30:38.248659 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.248530 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ebdd345-9423-4be2-8003-ac98c2c8d4c8" containerName="registry-server" Jul 07 18:30:38.248659 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.248542 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ebdd345-9423-4be2-8003-ac98c2c8d4c8" containerName="registry-server" Jul 07 18:30:38.248659 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.248590 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ebdd345-9423-4be2-8003-ac98c2c8d4c8" containerName="registry-server" Jul 07 18:30:38.251538 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.251523 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.259499 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.259471 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 18:30:38.259499 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.259495 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-sh5tt\"" Jul 07 18:30:38.259718 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.259592 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 18:30:38.259718 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.259625 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 18:30:38.259718 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.259703 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 18:30:38.259855 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.259742 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 18:30:38.259855 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.259745 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 18:30:38.259941 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.259930 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 18:30:38.266145 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.266124 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 18:30:38.270373 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.270351 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57f5bb7b7f-8mfgl"] Jul 07 18:30:38.313510 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.313471 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a7202c3-0e70-4409-a04c-6af1caa39305-console-config\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.313510 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.313517 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a7202c3-0e70-4409-a04c-6af1caa39305-oauth-serving-cert\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.313784 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.313557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7202c3-0e70-4409-a04c-6af1caa39305-service-ca\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.313784 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.313583 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a7202c3-0e70-4409-a04c-6af1caa39305-console-serving-cert\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.313784 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.313606 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a7202c3-0e70-4409-a04c-6af1caa39305-console-oauth-config\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.313784 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.313630 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a7202c3-0e70-4409-a04c-6af1caa39305-trusted-ca-bundle\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.313784 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.313689 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr6ld\" (UniqueName: \"kubernetes.io/projected/2a7202c3-0e70-4409-a04c-6af1caa39305-kube-api-access-pr6ld\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.342174 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.342136 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:38.414667 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.414629 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7202c3-0e70-4409-a04c-6af1caa39305-service-ca\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.414667 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.414670 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a7202c3-0e70-4409-a04c-6af1caa39305-console-serving-cert\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.414864 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.414687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a7202c3-0e70-4409-a04c-6af1caa39305-console-oauth-config\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.414864 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.414701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a7202c3-0e70-4409-a04c-6af1caa39305-trusted-ca-bundle\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.414864 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.414735 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr6ld\" (UniqueName: \"kubernetes.io/projected/2a7202c3-0e70-4409-a04c-6af1caa39305-kube-api-access-pr6ld\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.414864 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.414766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a7202c3-0e70-4409-a04c-6af1caa39305-console-config\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.414864 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.414789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a7202c3-0e70-4409-a04c-6af1caa39305-oauth-serving-cert\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.415460 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.415408 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7202c3-0e70-4409-a04c-6af1caa39305-service-ca\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.415557 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.415526 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a7202c3-0e70-4409-a04c-6af1caa39305-console-config\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.415624 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.415603 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a7202c3-0e70-4409-a04c-6af1caa39305-oauth-serving-cert\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.415686 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.415671 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a7202c3-0e70-4409-a04c-6af1caa39305-trusted-ca-bundle\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.417117 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.417095 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a7202c3-0e70-4409-a04c-6af1caa39305-console-serving-cert\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.417222 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.417206 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a7202c3-0e70-4409-a04c-6af1caa39305-console-oauth-config\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.428013 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.427986 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr6ld\" (UniqueName: \"kubernetes.io/projected/2a7202c3-0e70-4409-a04c-6af1caa39305-kube-api-access-pr6ld\") pod \"console-57f5bb7b7f-8mfgl\" (UID: \"2a7202c3-0e70-4409-a04c-6af1caa39305\") " pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.453276 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.453252 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:38.556044 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.555964 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:38.566018 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.565994 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:38.717695 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.717662 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57f5bb7b7f-8mfgl"] Jul 07 18:30:38.721588 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:30:38.721563 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7202c3_0e70_4409_a04c_6af1caa39305.slice/crio-61deee30570eb63f682b11b8f87cf195f55458a292923504f8fca619349043c8 WatchSource:0}: Error finding container 61deee30570eb63f682b11b8f87cf195f55458a292923504f8fca619349043c8: Status 404 returned error can't find the container with id 61deee30570eb63f682b11b8f87cf195f55458a292923504f8fca619349043c8 Jul 07 18:30:38.889296 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.889218 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f5bb7b7f-8mfgl" event={"ID":"2a7202c3-0e70-4409-a04c-6af1caa39305","Type":"ContainerStarted","Data":"507aff4edf043a9da7667a8e2c0c2271ce66c85e11705ad74fdde6f620c05b40"} Jul 07 18:30:38.889296 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.889267 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f5bb7b7f-8mfgl" event={"ID":"2a7202c3-0e70-4409-a04c-6af1caa39305","Type":"ContainerStarted","Data":"61deee30570eb63f682b11b8f87cf195f55458a292923504f8fca619349043c8"} Jul 07 18:30:38.909541 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:38.909491 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57f5bb7b7f-8mfgl" podStartSLOduration=0.90947533 podStartE2EDuration="909.47533ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:30:38.909147275 +0000 UTC m=+462.292940670" watchObservedRunningTime="2026-07-07 18:30:38.90947533 +0000 UTC m=+462.293268725" Jul 07 18:30:39.073877 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:39.073844 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:39.212871 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:39.212798 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:43.960093 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:43.960063 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:44.021478 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:44.021445 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:45.991029 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:45.990996 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:46.092388 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:46.092359 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:48.566268 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:48.566224 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:48.566779 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:48.566299 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:48.571107 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:48.571082 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:48.932657 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:48.932580 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57f5bb7b7f-8mfgl" Jul 07 18:30:49.113625 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:49.113517 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:49.214704 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:49.214617 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:49.332737 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:49.332702 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:52.333533 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:52.333502 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:52.940770 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:52.940738 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:54.163045 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:54.163012 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:54.259100 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:54.259064 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:56.379362 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:56.379314 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:56.541777 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:56.541748 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-96qch"] Jul 07 18:30:56.548262 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:56.548236 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-96qch" Jul 07 18:30:56.551231 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:56.551213 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-zs7dl\"" Jul 07 18:30:56.555936 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:56.555910 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-96qch"] Jul 07 18:30:56.658582 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:56.658491 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnrdh\" (UniqueName: \"kubernetes.io/projected/71414372-dc1a-42f8-8711-e24686b4f505-kube-api-access-fnrdh\") pod \"authorino-operator-75cc8d5cb7-96qch\" (UID: \"71414372-dc1a-42f8-8711-e24686b4f505\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-96qch" Jul 07 18:30:56.758909 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:56.758879 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnrdh\" (UniqueName: \"kubernetes.io/projected/71414372-dc1a-42f8-8711-e24686b4f505-kube-api-access-fnrdh\") pod \"authorino-operator-75cc8d5cb7-96qch\" (UID: \"71414372-dc1a-42f8-8711-e24686b4f505\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-96qch" Jul 07 18:30:56.768280 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:56.768247 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnrdh\" (UniqueName: \"kubernetes.io/projected/71414372-dc1a-42f8-8711-e24686b4f505-kube-api-access-fnrdh\") pod \"authorino-operator-75cc8d5cb7-96qch\" (UID: \"71414372-dc1a-42f8-8711-e24686b4f505\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-96qch" Jul 07 18:30:56.859294 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:56.859258 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-96qch" Jul 07 18:30:56.991040 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:30:56.991003 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71414372_dc1a_42f8_8711_e24686b4f505.slice/crio-cc437b70fb6e403ed0c6dc2c947ceb0d7300ddb744b0bc54079559bf7159dd78 WatchSource:0}: Error finding container cc437b70fb6e403ed0c6dc2c947ceb0d7300ddb744b0bc54079559bf7159dd78: Status 404 returned error can't find the container with id cc437b70fb6e403ed0c6dc2c947ceb0d7300ddb744b0bc54079559bf7159dd78 Jul 07 18:30:56.991597 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:56.991575 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-96qch"] Jul 07 18:30:57.578518 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:57.578490 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:57.965964 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:57.965886 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-96qch" event={"ID":"71414372-dc1a-42f8-8711-e24686b4f505","Type":"ContainerStarted","Data":"cc437b70fb6e403ed0c6dc2c947ceb0d7300ddb744b0bc54079559bf7159dd78"} Jul 07 18:30:59.305148 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:59.305109 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq"] Jul 07 18:30:59.308542 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:59.308522 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" Jul 07 18:30:59.311844 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:59.311815 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-m6nk5\"" Jul 07 18:30:59.331165 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:59.331140 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq"] Jul 07 18:30:59.377225 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:59.377191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57kvb\" (UniqueName: \"kubernetes.io/projected/12ec1f44-ac0f-4afb-9c7e-bf53970ee365-kube-api-access-57kvb\") pod \"limitador-operator-controller-manager-859d7b78bd-rzmwq\" (UID: \"12ec1f44-ac0f-4afb-9c7e-bf53970ee365\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" Jul 07 18:30:59.478168 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:59.478130 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57kvb\" (UniqueName: \"kubernetes.io/projected/12ec1f44-ac0f-4afb-9c7e-bf53970ee365-kube-api-access-57kvb\") pod \"limitador-operator-controller-manager-859d7b78bd-rzmwq\" (UID: \"12ec1f44-ac0f-4afb-9c7e-bf53970ee365\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" Jul 07 18:30:59.488354 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:59.488304 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57kvb\" (UniqueName: \"kubernetes.io/projected/12ec1f44-ac0f-4afb-9c7e-bf53970ee365-kube-api-access-57kvb\") pod \"limitador-operator-controller-manager-859d7b78bd-rzmwq\" (UID: \"12ec1f44-ac0f-4afb-9c7e-bf53970ee365\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" Jul 07 18:30:59.619379 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:59.619276 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" Jul 07 18:30:59.745698 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:59.745673 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:59.776152 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:59.776034 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq"] Jul 07 18:30:59.778308 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:30:59.778279 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ec1f44_ac0f_4afb_9c7e_bf53970ee365.slice/crio-dd5f2342cf2c7aeb30d71d2728d074adb7c971387326e37dd573cb53c200328d WatchSource:0}: Error finding container dd5f2342cf2c7aeb30d71d2728d074adb7c971387326e37dd573cb53c200328d: Status 404 returned error can't find the container with id dd5f2342cf2c7aeb30d71d2728d074adb7c971387326e37dd573cb53c200328d Jul 07 18:30:59.976239 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:59.976207 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" event={"ID":"12ec1f44-ac0f-4afb-9c7e-bf53970ee365","Type":"ContainerStarted","Data":"dd5f2342cf2c7aeb30d71d2728d074adb7c971387326e37dd573cb53c200328d"} Jul 07 18:30:59.977561 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:59.977537 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-96qch" event={"ID":"71414372-dc1a-42f8-8711-e24686b4f505","Type":"ContainerStarted","Data":"c8b09d6133eb3fef7886bbd32c73a472100bc738b9b19f7f34a237ae4e0fda8e"} Jul 07 18:30:59.977694 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:59.977677 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-96qch" Jul 07 18:30:59.999562 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:30:59.999521 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-96qch" podStartSLOduration=1.344261591 podStartE2EDuration="3.999505989s" totalImagesPullingTime="2.655244398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:30:59.998540318 +0000 UTC m=+483.382333715" watchObservedRunningTime="2026-07-07 18:30:59.999505989 +0000 UTC m=+483.383299383" Jul 07 18:31:00.600978 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:00.600945 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:01.986734 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:01.986697 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" event={"ID":"12ec1f44-ac0f-4afb-9c7e-bf53970ee365","Type":"ContainerStarted","Data":"b8eb182af33841d4935af02920cf2d6349d585afb74b8923167ee7251768d488"} Jul 07 18:31:01.987108 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:01.986768 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" Jul 07 18:31:02.003391 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:02.003365 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:02.013356 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:02.013297 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" podStartSLOduration=1.106145609 podStartE2EDuration="3.013277927s" totalImagesPullingTime="1.907132318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:31:02.011835774 +0000 UTC m=+485.395629175" watchObservedRunningTime="2026-07-07 18:31:02.013277927 +0000 UTC m=+485.397071322" Jul 07 18:31:03.035939 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:03.035909 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8"] Jul 07 18:31:03.039125 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:03.039109 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:03.041648 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:03.041629 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p2v2g\"" Jul 07 18:31:03.056917 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:03.056895 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8"] Jul 07 18:31:03.108660 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:03.108627 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/709527d4-3e82-486d-b09d-60e8b8df4877-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kdlh8\" (UID: \"709527d4-3e82-486d-b09d-60e8b8df4877\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:03.108809 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:03.108678 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z8np\" (UniqueName: \"kubernetes.io/projected/709527d4-3e82-486d-b09d-60e8b8df4877-kube-api-access-2z8np\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kdlh8\" (UID: \"709527d4-3e82-486d-b09d-60e8b8df4877\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:03.210042 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:03.210016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/709527d4-3e82-486d-b09d-60e8b8df4877-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kdlh8\" (UID: \"709527d4-3e82-486d-b09d-60e8b8df4877\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:03.210199 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:03.210056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z8np\" (UniqueName: \"kubernetes.io/projected/709527d4-3e82-486d-b09d-60e8b8df4877-kube-api-access-2z8np\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kdlh8\" (UID: \"709527d4-3e82-486d-b09d-60e8b8df4877\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:03.210431 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:03.210411 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/709527d4-3e82-486d-b09d-60e8b8df4877-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kdlh8\" (UID: \"709527d4-3e82-486d-b09d-60e8b8df4877\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:03.244811 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:03.244783 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z8np\" (UniqueName: \"kubernetes.io/projected/709527d4-3e82-486d-b09d-60e8b8df4877-kube-api-access-2z8np\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kdlh8\" (UID: \"709527d4-3e82-486d-b09d-60e8b8df4877\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:03.348555 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:03.348463 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:03.482944 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:03.482920 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8"] Jul 07 18:31:03.485507 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:31:03.485480 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709527d4_3e82_486d_b09d_60e8b8df4877.slice/crio-359c6f1aa9bb290f645f4535d47704f99bec482801ff7a58f073699b51ae96c9 WatchSource:0}: Error finding container 359c6f1aa9bb290f645f4535d47704f99bec482801ff7a58f073699b51ae96c9: Status 404 returned error can't find the container with id 359c6f1aa9bb290f645f4535d47704f99bec482801ff7a58f073699b51ae96c9 Jul 07 18:31:03.578596 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:03.578562 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:03.995467 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:03.995435 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" event={"ID":"709527d4-3e82-486d-b09d-60e8b8df4877","Type":"ContainerStarted","Data":"359c6f1aa9bb290f645f4535d47704f99bec482801ff7a58f073699b51ae96c9"} Jul 07 18:31:04.192172 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:04.192141 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:05.204010 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:05.203984 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcjlp"] Jul 07 18:31:05.206946 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:05.206931 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcjlp" Jul 07 18:31:05.210931 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:05.210905 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 07 18:31:05.211053 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:05.210986 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2gbgh\"" Jul 07 18:31:05.222867 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:05.222847 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcjlp"] Jul 07 18:31:05.326951 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:05.326918 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfxvf\" (UniqueName: \"kubernetes.io/projected/67cece42-5d7a-41d1-adf0-10579cdb66bb-kube-api-access-cfxvf\") pod \"dns-operator-controller-manager-864dd78c4c-pcjlp\" (UID: \"67cece42-5d7a-41d1-adf0-10579cdb66bb\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcjlp" Jul 07 18:31:05.427286 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:05.427258 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfxvf\" (UniqueName: \"kubernetes.io/projected/67cece42-5d7a-41d1-adf0-10579cdb66bb-kube-api-access-cfxvf\") pod \"dns-operator-controller-manager-864dd78c4c-pcjlp\" (UID: \"67cece42-5d7a-41d1-adf0-10579cdb66bb\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcjlp" Jul 07 18:31:05.436838 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:05.436812 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfxvf\" (UniqueName: \"kubernetes.io/projected/67cece42-5d7a-41d1-adf0-10579cdb66bb-kube-api-access-cfxvf\") pod \"dns-operator-controller-manager-864dd78c4c-pcjlp\" (UID: \"67cece42-5d7a-41d1-adf0-10579cdb66bb\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcjlp" Jul 07 18:31:05.517129 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:05.517093 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcjlp" Jul 07 18:31:05.654685 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:05.654652 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcjlp"] Jul 07 18:31:05.656699 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:31:05.656674 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67cece42_5d7a_41d1_adf0_10579cdb66bb.slice/crio-aa938965c32d71597902732cca405ec99eae039ea30decf2b5447711e0796ee0 WatchSource:0}: Error finding container aa938965c32d71597902732cca405ec99eae039ea30decf2b5447711e0796ee0: Status 404 returned error can't find the container with id aa938965c32d71597902732cca405ec99eae039ea30decf2b5447711e0796ee0 Jul 07 18:31:06.005217 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:06.005183 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcjlp" event={"ID":"67cece42-5d7a-41d1-adf0-10579cdb66bb","Type":"ContainerStarted","Data":"aa938965c32d71597902732cca405ec99eae039ea30decf2b5447711e0796ee0"} Jul 07 18:31:06.554661 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:06.554629 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:06.656564 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:06.656529 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:06.946769 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:06.946703 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:07.063702 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:07.063644 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:08.016634 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:08.016597 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcjlp" event={"ID":"67cece42-5d7a-41d1-adf0-10579cdb66bb","Type":"ContainerStarted","Data":"31296822659a667c9ff99b0765908d40c618790410ecb20561b770b745224fd9"} Jul 07 18:31:08.016990 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:08.016705 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcjlp" Jul 07 18:31:08.037379 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:08.037314 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcjlp" podStartSLOduration=0.767434567 podStartE2EDuration="3.037292697s" totalImagesPullingTime="2.26985813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:31:08.035854176 +0000 UTC m=+491.419647623" watchObservedRunningTime="2026-07-07 18:31:08.037292697 +0000 UTC m=+491.421086097" Jul 07 18:31:10.366650 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:10.366618 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:10.527433 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:10.527403 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:10.983522 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:10.983494 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-96qch" Jul 07 18:31:11.033181 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:11.033147 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" event={"ID":"709527d4-3e82-486d-b09d-60e8b8df4877","Type":"ContainerStarted","Data":"f4bfa99dece99c944d0538b8cea59c662c9c42b6b533633b93b109fc6caac8de"} Jul 07 18:31:11.033348 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:11.033233 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:11.054798 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:11.054748 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" podStartSLOduration=1.329705194 podStartE2EDuration="8.054727581s" totalImagesPullingTime="6.725022387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:31:11.054061522 +0000 UTC m=+494.437854917" watchObservedRunningTime="2026-07-07 18:31:11.054727581 +0000 UTC m=+494.438520977" Jul 07 18:31:12.992384 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:12.992348 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" Jul 07 18:31:13.017821 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:13.017790 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:13.361067 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:13.360984 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:13.693541 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:13.693461 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:13.954799 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:13.954704 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:14.751753 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:14.751721 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:14.848579 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:14.848546 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:19.024313 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:19.024279 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pcjlp" Jul 07 18:31:20.056692 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:20.056659 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:20.118990 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:20.118954 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:22.040252 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:22.040222 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:23.458536 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.458502 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp"] Jul 07 18:31:23.461804 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.461781 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp" Jul 07 18:31:23.478493 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.478464 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp"] Jul 07 18:31:23.583238 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.583197 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvdgk\" (UniqueName: \"kubernetes.io/projected/727f9eb4-e203-4dc8-8f73-ee6beaa3360c-kube-api-access-pvdgk\") pod \"kuadrant-operator-controller-manager-55467d9f7-9gclp\" (UID: \"727f9eb4-e203-4dc8-8f73-ee6beaa3360c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp" Jul 07 18:31:23.583419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.583258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/727f9eb4-e203-4dc8-8f73-ee6beaa3360c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-9gclp\" (UID: \"727f9eb4-e203-4dc8-8f73-ee6beaa3360c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp" Jul 07 18:31:23.684370 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.684314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvdgk\" (UniqueName: \"kubernetes.io/projected/727f9eb4-e203-4dc8-8f73-ee6beaa3360c-kube-api-access-pvdgk\") pod \"kuadrant-operator-controller-manager-55467d9f7-9gclp\" (UID: \"727f9eb4-e203-4dc8-8f73-ee6beaa3360c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp" Jul 07 18:31:23.684576 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.684399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/727f9eb4-e203-4dc8-8f73-ee6beaa3360c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-9gclp\" (UID: \"727f9eb4-e203-4dc8-8f73-ee6beaa3360c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp" Jul 07 18:31:23.684768 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.684743 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/727f9eb4-e203-4dc8-8f73-ee6beaa3360c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-9gclp\" (UID: \"727f9eb4-e203-4dc8-8f73-ee6beaa3360c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp" Jul 07 18:31:23.698120 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.698076 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvdgk\" (UniqueName: \"kubernetes.io/projected/727f9eb4-e203-4dc8-8f73-ee6beaa3360c-kube-api-access-pvdgk\") pod \"kuadrant-operator-controller-manager-55467d9f7-9gclp\" (UID: \"727f9eb4-e203-4dc8-8f73-ee6beaa3360c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp" Jul 07 18:31:23.771782 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.771740 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp" Jul 07 18:31:23.852077 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.852039 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp"] Jul 07 18:31:23.859194 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.859161 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp"] Jul 07 18:31:23.875385 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.875356 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz"] Jul 07 18:31:23.878764 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.878741 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz" Jul 07 18:31:23.893403 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.893374 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz"] Jul 07 18:31:23.897573 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.897543 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8"] Jul 07 18:31:23.897840 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.897790 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" podUID="709527d4-3e82-486d-b09d-60e8b8df4877" containerName="manager" containerID="cri-o://f4bfa99dece99c944d0538b8cea59c662c9c42b6b533633b93b109fc6caac8de" gracePeriod=2 Jul 07 18:31:23.920518 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.920489 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8"] Jul 07 18:31:23.944545 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.944509 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq"] Jul 07 18:31:23.946033 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.945024 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" podUID="12ec1f44-ac0f-4afb-9c7e-bf53970ee365" containerName="manager" containerID="cri-o://b8eb182af33841d4935af02920cf2d6349d585afb74b8923167ee7251768d488" gracePeriod=2 Jul 07 18:31:23.946942 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:23.946895 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kdlh8\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="709527d4-3e82-486d-b09d-60e8b8df4877" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:23.958019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.957984 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq"] Jul 07 18:31:23.966044 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:23.966003 2578 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"12ec1f44-ac0f-4afb-9c7e-bf53970ee365\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-rzmwq\": pods \"limitador-operator-controller-manager-859d7b78bd-rzmwq\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" Jul 07 18:31:23.971846 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.971813 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9f846"] Jul 07 18:31:23.972286 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.972270 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12ec1f44-ac0f-4afb-9c7e-bf53970ee365" containerName="manager" Jul 07 18:31:23.972385 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.972290 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="12ec1f44-ac0f-4afb-9c7e-bf53970ee365" containerName="manager" Jul 07 18:31:23.972385 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.972305 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="709527d4-3e82-486d-b09d-60e8b8df4877" containerName="manager" Jul 07 18:31:23.972834 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.972313 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="709527d4-3e82-486d-b09d-60e8b8df4877" containerName="manager" Jul 07 18:31:23.972949 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.972935 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="12ec1f44-ac0f-4afb-9c7e-bf53970ee365" containerName="manager" Jul 07 18:31:23.973001 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.972957 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="709527d4-3e82-486d-b09d-60e8b8df4877" containerName="manager" Jul 07 18:31:23.976054 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.976034 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9f846" Jul 07 18:31:23.978339 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:23.978296 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kdlh8\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="709527d4-3e82-486d-b09d-60e8b8df4877" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:23.980417 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:23.980382 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rzmwq\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="12ec1f44-ac0f-4afb-9c7e-bf53970ee365" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" Jul 07 18:31:23.986423 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.986394 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs8mx\" (UniqueName: \"kubernetes.io/projected/99bdfb3a-2c41-47cc-825b-4209cb5230c5-kube-api-access-gs8mx\") pod \"kuadrant-operator-controller-manager-55467d9f7-s6fpz\" (UID: \"99bdfb3a-2c41-47cc-825b-4209cb5230c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz" Jul 07 18:31:23.986513 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.986486 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/99bdfb3a-2c41-47cc-825b-4209cb5230c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-s6fpz\" (UID: \"99bdfb3a-2c41-47cc-825b-4209cb5230c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz" Jul 07 18:31:23.990426 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:23.990399 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9f846"] Jul 07 18:31:24.047521 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:24.047491 2578 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 07 18:31:24.047521 ip-10-0-141-108 kubenswrapper[2578]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-9gclp_kuadrant-system_727f9eb4-e203-4dc8-8f73-ee6beaa3360c_0(0017af4e6c0b04868dfe08c2a6ba70289b304938450ca73055328f97ff636e28): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-9gclp to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"0017af4e6c0b04868dfe08c2a6ba70289b304938450ca73055328f97ff636e28" Netns:"/var/run/netns/62ebd9d6-8bcc-4522-b5f5-584255955977" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-55467d9f7-9gclp;K8S_POD_INFRA_CONTAINER_ID=0017af4e6c0b04868dfe08c2a6ba70289b304938450ca73055328f97ff636e28;K8S_POD_UID=727f9eb4-e203-4dc8-8f73-ee6beaa3360c" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp] networking: [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp/727f9eb4-e203-4dc8-8f73-ee6beaa3360c:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp 0017af4e6c0b04868dfe08c2a6ba70289b304938450ca73055328f97ff636e28 network default NAD default NAD key default] [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp 0017af4e6c0b04868dfe08c2a6ba70289b304938450ca73055328f97ff636e28 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:86:00:1c [10.134.0.28/23] Jul 07 18:31:24.047521 ip-10-0-141-108 kubenswrapper[2578]: ' Jul 07 18:31:24.047521 ip-10-0-141-108 kubenswrapper[2578]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 07 18:31:24.047521 ip-10-0-141-108 kubenswrapper[2578]: > Jul 07 18:31:24.047725 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:24.047546 2578 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 07 18:31:24.047725 ip-10-0-141-108 kubenswrapper[2578]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-9gclp_kuadrant-system_727f9eb4-e203-4dc8-8f73-ee6beaa3360c_0(0017af4e6c0b04868dfe08c2a6ba70289b304938450ca73055328f97ff636e28): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-9gclp to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"0017af4e6c0b04868dfe08c2a6ba70289b304938450ca73055328f97ff636e28" Netns:"/var/run/netns/62ebd9d6-8bcc-4522-b5f5-584255955977" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-55467d9f7-9gclp;K8S_POD_INFRA_CONTAINER_ID=0017af4e6c0b04868dfe08c2a6ba70289b304938450ca73055328f97ff636e28;K8S_POD_UID=727f9eb4-e203-4dc8-8f73-ee6beaa3360c" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp] networking: [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp/727f9eb4-e203-4dc8-8f73-ee6beaa3360c:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp 0017af4e6c0b04868dfe08c2a6ba70289b304938450ca73055328f97ff636e28 network default NAD default NAD key default] [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp 0017af4e6c0b04868dfe08c2a6ba70289b304938450ca73055328f97ff636e28 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:86:00:1c [10.134.0.28/23] Jul 07 18:31:24.047725 ip-10-0-141-108 kubenswrapper[2578]: ' Jul 07 18:31:24.047725 ip-10-0-141-108 kubenswrapper[2578]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 07 18:31:24.047725 ip-10-0-141-108 kubenswrapper[2578]: > pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp" Jul 07 18:31:24.087691 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.087085 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/99bdfb3a-2c41-47cc-825b-4209cb5230c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-s6fpz\" (UID: \"99bdfb3a-2c41-47cc-825b-4209cb5230c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz" Jul 07 18:31:24.087691 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.087186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgqtr\" (UniqueName: \"kubernetes.io/projected/12a67f8c-909f-49c0-960f-2f8c51692968-kube-api-access-zgqtr\") pod \"limitador-operator-controller-manager-859d7b78bd-9f846\" (UID: \"12a67f8c-909f-49c0-960f-2f8c51692968\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9f846" Jul 07 18:31:24.087691 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.087247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs8mx\" (UniqueName: \"kubernetes.io/projected/99bdfb3a-2c41-47cc-825b-4209cb5230c5-kube-api-access-gs8mx\") pod \"kuadrant-operator-controller-manager-55467d9f7-s6fpz\" (UID: \"99bdfb3a-2c41-47cc-825b-4209cb5230c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz" Jul 07 18:31:24.087691 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.087503 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/99bdfb3a-2c41-47cc-825b-4209cb5230c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-s6fpz\" (UID: \"99bdfb3a-2c41-47cc-825b-4209cb5230c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz" Jul 07 18:31:24.092923 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.092889 2578 generic.go:356] "Generic (PLEG): container finished" podID="12ec1f44-ac0f-4afb-9c7e-bf53970ee365" containerID="b8eb182af33841d4935af02920cf2d6349d585afb74b8923167ee7251768d488" exitCode=0 Jul 07 18:31:24.095392 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.095360 2578 generic.go:356] "Generic (PLEG): container finished" podID="709527d4-3e82-486d-b09d-60e8b8df4877" containerID="f4bfa99dece99c944d0538b8cea59c662c9c42b6b533633b93b109fc6caac8de" exitCode=0 Jul 07 18:31:24.095561 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.095436 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp" Jul 07 18:31:24.097098 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.097078 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs8mx\" (UniqueName: \"kubernetes.io/projected/99bdfb3a-2c41-47cc-825b-4209cb5230c5-kube-api-access-gs8mx\") pod \"kuadrant-operator-controller-manager-55467d9f7-s6fpz\" (UID: \"99bdfb3a-2c41-47cc-825b-4209cb5230c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz" Jul 07 18:31:24.097391 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:24.097363 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kdlh8\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="709527d4-3e82-486d-b09d-60e8b8df4877" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:24.099104 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:24.099071 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rzmwq\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="12ec1f44-ac0f-4afb-9c7e-bf53970ee365" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" Jul 07 18:31:24.100616 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:24.100578 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-9gclp\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="727f9eb4-e203-4dc8-8f73-ee6beaa3360c" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp" Jul 07 18:31:24.102949 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.102929 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp" Jul 07 18:31:24.104629 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:24.104610 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kdlh8\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="709527d4-3e82-486d-b09d-60e8b8df4877" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:24.106377 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:24.106345 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rzmwq\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="12ec1f44-ac0f-4afb-9c7e-bf53970ee365" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" Jul 07 18:31:24.107968 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:24.107946 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-9gclp\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="727f9eb4-e203-4dc8-8f73-ee6beaa3360c" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp" Jul 07 18:31:24.175335 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.175295 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:24.180194 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:24.180165 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kdlh8\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="709527d4-3e82-486d-b09d-60e8b8df4877" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:24.182082 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:24.182053 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rzmwq\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="12ec1f44-ac0f-4afb-9c7e-bf53970ee365" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" Jul 07 18:31:24.184802 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.184785 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" Jul 07 18:31:24.185508 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:24.185488 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-9gclp\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="727f9eb4-e203-4dc8-8f73-ee6beaa3360c" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp" Jul 07 18:31:24.187379 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:24.187358 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kdlh8\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="709527d4-3e82-486d-b09d-60e8b8df4877" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:24.187495 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.187478 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvdgk\" (UniqueName: \"kubernetes.io/projected/727f9eb4-e203-4dc8-8f73-ee6beaa3360c-kube-api-access-pvdgk\") pod \"727f9eb4-e203-4dc8-8f73-ee6beaa3360c\" (UID: \"727f9eb4-e203-4dc8-8f73-ee6beaa3360c\") " Jul 07 18:31:24.187662 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.187649 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/727f9eb4-e203-4dc8-8f73-ee6beaa3360c-extensions-socket-volume\") pod \"727f9eb4-e203-4dc8-8f73-ee6beaa3360c\" (UID: \"727f9eb4-e203-4dc8-8f73-ee6beaa3360c\") " Jul 07 18:31:24.187736 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.187722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgqtr\" (UniqueName: \"kubernetes.io/projected/12a67f8c-909f-49c0-960f-2f8c51692968-kube-api-access-zgqtr\") pod \"limitador-operator-controller-manager-859d7b78bd-9f846\" (UID: \"12a67f8c-909f-49c0-960f-2f8c51692968\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9f846" Jul 07 18:31:24.187915 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.187890 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727f9eb4-e203-4dc8-8f73-ee6beaa3360c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "727f9eb4-e203-4dc8-8f73-ee6beaa3360c" (UID: "727f9eb4-e203-4dc8-8f73-ee6beaa3360c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:31:24.189522 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.189503 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727f9eb4-e203-4dc8-8f73-ee6beaa3360c-kube-api-access-pvdgk" (OuterVolumeSpecName: "kube-api-access-pvdgk") pod "727f9eb4-e203-4dc8-8f73-ee6beaa3360c" (UID: "727f9eb4-e203-4dc8-8f73-ee6beaa3360c"). InnerVolumeSpecName "kube-api-access-pvdgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:31:24.190612 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.190595 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz" Jul 07 18:31:24.194248 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:24.194225 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rzmwq\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="12ec1f44-ac0f-4afb-9c7e-bf53970ee365" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" Jul 07 18:31:24.196731 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:24.196712 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-9gclp\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="727f9eb4-e203-4dc8-8f73-ee6beaa3360c" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp" Jul 07 18:31:24.200557 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.200314 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgqtr\" (UniqueName: \"kubernetes.io/projected/12a67f8c-909f-49c0-960f-2f8c51692968-kube-api-access-zgqtr\") pod \"limitador-operator-controller-manager-859d7b78bd-9f846\" (UID: \"12a67f8c-909f-49c0-960f-2f8c51692968\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9f846" Jul 07 18:31:24.288785 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.288755 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/709527d4-3e82-486d-b09d-60e8b8df4877-extensions-socket-volume\") pod \"709527d4-3e82-486d-b09d-60e8b8df4877\" (UID: \"709527d4-3e82-486d-b09d-60e8b8df4877\") " Jul 07 18:31:24.288945 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.288807 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57kvb\" (UniqueName: \"kubernetes.io/projected/12ec1f44-ac0f-4afb-9c7e-bf53970ee365-kube-api-access-57kvb\") pod \"12ec1f44-ac0f-4afb-9c7e-bf53970ee365\" (UID: \"12ec1f44-ac0f-4afb-9c7e-bf53970ee365\") " Jul 07 18:31:24.288945 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.288837 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z8np\" (UniqueName: \"kubernetes.io/projected/709527d4-3e82-486d-b09d-60e8b8df4877-kube-api-access-2z8np\") pod \"709527d4-3e82-486d-b09d-60e8b8df4877\" (UID: \"709527d4-3e82-486d-b09d-60e8b8df4877\") " Jul 07 18:31:24.289040 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.289027 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/727f9eb4-e203-4dc8-8f73-ee6beaa3360c-extensions-socket-volume\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:31:24.289082 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.289042 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvdgk\" (UniqueName: \"kubernetes.io/projected/727f9eb4-e203-4dc8-8f73-ee6beaa3360c-kube-api-access-pvdgk\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:31:24.289379 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.289352 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709527d4-3e82-486d-b09d-60e8b8df4877-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "709527d4-3e82-486d-b09d-60e8b8df4877" (UID: "709527d4-3e82-486d-b09d-60e8b8df4877"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:31:24.291287 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.291263 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709527d4-3e82-486d-b09d-60e8b8df4877-kube-api-access-2z8np" (OuterVolumeSpecName: "kube-api-access-2z8np") pod "709527d4-3e82-486d-b09d-60e8b8df4877" (UID: "709527d4-3e82-486d-b09d-60e8b8df4877"). InnerVolumeSpecName "kube-api-access-2z8np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:31:24.291411 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.291314 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ec1f44-ac0f-4afb-9c7e-bf53970ee365-kube-api-access-57kvb" (OuterVolumeSpecName: "kube-api-access-57kvb") pod "12ec1f44-ac0f-4afb-9c7e-bf53970ee365" (UID: "12ec1f44-ac0f-4afb-9c7e-bf53970ee365"). InnerVolumeSpecName "kube-api-access-57kvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:31:24.311557 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.311477 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9f846" Jul 07 18:31:24.340431 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.340287 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz"] Jul 07 18:31:24.342758 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:31:24.342728 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99bdfb3a_2c41_47cc_825b_4209cb5230c5.slice/crio-9b69c1a9b736cb83edace16a39b7c70aa7e4ef7b91da77b5879635d6750211ce WatchSource:0}: Error finding container 9b69c1a9b736cb83edace16a39b7c70aa7e4ef7b91da77b5879635d6750211ce: Status 404 returned error can't find the container with id 9b69c1a9b736cb83edace16a39b7c70aa7e4ef7b91da77b5879635d6750211ce Jul 07 18:31:24.389604 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.389577 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57kvb\" (UniqueName: \"kubernetes.io/projected/12ec1f44-ac0f-4afb-9c7e-bf53970ee365-kube-api-access-57kvb\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:31:24.389604 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.389603 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z8np\" (UniqueName: \"kubernetes.io/projected/709527d4-3e82-486d-b09d-60e8b8df4877-kube-api-access-2z8np\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:31:24.389774 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.389616 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/709527d4-3e82-486d-b09d-60e8b8df4877-extensions-socket-volume\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:31:24.459299 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:24.459127 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9f846"] Jul 07 18:31:24.461964 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:31:24.461934 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a67f8c_909f_49c0_960f_2f8c51692968.slice/crio-b7a672beaea60420b6a4eb15ba6e5f54257b621d3ff6fdd2b6b4d1644ab353ab WatchSource:0}: Error finding container b7a672beaea60420b6a4eb15ba6e5f54257b621d3ff6fdd2b6b4d1644ab353ab: Status 404 returned error can't find the container with id b7a672beaea60420b6a4eb15ba6e5f54257b621d3ff6fdd2b6b4d1644ab353ab Jul 07 18:31:25.100983 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:25.100957 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" Jul 07 18:31:25.100983 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:25.100965 2578 scope.go:117] "RemoveContainer" containerID="b8eb182af33841d4935af02920cf2d6349d585afb74b8923167ee7251768d488" Jul 07 18:31:25.102728 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:25.102690 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz" event={"ID":"99bdfb3a-2c41-47cc-825b-4209cb5230c5","Type":"ContainerStarted","Data":"f1d75bdf49b84ee1c3477ffee6574c4f359ed278c809475207fc2f3b87a62695"} Jul 07 18:31:25.102848 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:25.102747 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz" Jul 07 18:31:25.102848 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:25.102773 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz" event={"ID":"99bdfb3a-2c41-47cc-825b-4209cb5230c5","Type":"ContainerStarted","Data":"9b69c1a9b736cb83edace16a39b7c70aa7e4ef7b91da77b5879635d6750211ce"} Jul 07 18:31:25.103361 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:25.103310 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kdlh8\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="709527d4-3e82-486d-b09d-60e8b8df4877" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:25.104129 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:25.104105 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:25.105157 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:25.105126 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rzmwq\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="12ec1f44-ac0f-4afb-9c7e-bf53970ee365" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" Jul 07 18:31:25.105862 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:25.105843 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp" Jul 07 18:31:25.105956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:25.105882 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9f846" event={"ID":"12a67f8c-909f-49c0-960f-2f8c51692968","Type":"ContainerStarted","Data":"c4ee4654bf2411022c098bf642c1e2ce7451aa61d2f318e834488cf36d46b590"} Jul 07 18:31:25.105956 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:25.105930 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9f846" event={"ID":"12a67f8c-909f-49c0-960f-2f8c51692968","Type":"ContainerStarted","Data":"b7a672beaea60420b6a4eb15ba6e5f54257b621d3ff6fdd2b6b4d1644ab353ab"} Jul 07 18:31:25.106057 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:25.106013 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9f846" Jul 07 18:31:25.106716 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:25.106694 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-9gclp\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="727f9eb4-e203-4dc8-8f73-ee6beaa3360c" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp" Jul 07 18:31:25.108434 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:25.108408 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rzmwq\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="12ec1f44-ac0f-4afb-9c7e-bf53970ee365" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" Jul 07 18:31:25.110038 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:25.110005 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-9gclp\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="727f9eb4-e203-4dc8-8f73-ee6beaa3360c" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp" Jul 07 18:31:25.110153 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:25.110060 2578 scope.go:117] "RemoveContainer" containerID="f4bfa99dece99c944d0538b8cea59c662c9c42b6b533633b93b109fc6caac8de" Jul 07 18:31:25.131153 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:25.131116 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9f846" podStartSLOduration=2.131102251 podStartE2EDuration="2.131102251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:31:25.129170652 +0000 UTC m=+508.512964060" watchObservedRunningTime="2026-07-07 18:31:25.131102251 +0000 UTC m=+508.514895689" Jul 07 18:31:25.149451 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:25.149411 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz" podStartSLOduration=2.149396872 podStartE2EDuration="2.149396872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:31:25.147816756 +0000 UTC m=+508.531610162" watchObservedRunningTime="2026-07-07 18:31:25.149396872 +0000 UTC m=+508.533190266" Jul 07 18:31:25.150538 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:25.150514 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kdlh8\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="709527d4-3e82-486d-b09d-60e8b8df4877" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:25.152435 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:25.152415 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kdlh8\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="709527d4-3e82-486d-b09d-60e8b8df4877" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kdlh8" Jul 07 18:31:25.154263 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:25.154227 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rzmwq\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="12ec1f44-ac0f-4afb-9c7e-bf53970ee365" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzmwq" Jul 07 18:31:25.155900 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:25.155880 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-9gclp\" is forbidden: User \"system:node:ip-10-0-141-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-108.ec2.internal' and this object" podUID="727f9eb4-e203-4dc8-8f73-ee6beaa3360c" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9gclp" Jul 07 18:31:25.201705 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:25.199357 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12ec1f44-ac0f-4afb-9c7e-bf53970ee365" path="/var/lib/kubelet/pods/12ec1f44-ac0f-4afb-9c7e-bf53970ee365/volumes" Jul 07 18:31:25.201705 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:25.199859 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="709527d4-3e82-486d-b09d-60e8b8df4877" path="/var/lib/kubelet/pods/709527d4-3e82-486d-b09d-60e8b8df4877/volumes" Jul 07 18:31:25.201705 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:25.200300 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="727f9eb4-e203-4dc8-8f73-ee6beaa3360c" path="/var/lib/kubelet/pods/727f9eb4-e203-4dc8-8f73-ee6beaa3360c/volumes" Jul 07 18:31:25.809489 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:25.809455 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:25.907596 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:25.907566 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:26.459615 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:26.459580 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:26.560078 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:26.560045 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:29.791393 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:29.791364 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:29.892859 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:29.892828 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:33.484678 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:33.484648 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:33.584218 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:33.584188 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:36.113792 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:36.113769 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9f846" Jul 07 18:31:36.114188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:36.113828 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz" Jul 07 18:31:36.611910 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:36.611883 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26"] Jul 07 18:31:36.661501 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:36.661475 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26"] Jul 07 18:31:36.661662 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:36.661574 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26" Jul 07 18:31:36.778127 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:36.778030 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctkjf\" (UniqueName: \"kubernetes.io/projected/c314533e-8a11-40dd-8549-77e5aa5e42a7-kube-api-access-ctkjf\") pod \"kuadrant-operator-controller-manager-86878c769f-kkw26\" (UID: \"c314533e-8a11-40dd-8549-77e5aa5e42a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26" Jul 07 18:31:36.778127 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:36.778070 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c314533e-8a11-40dd-8549-77e5aa5e42a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-kkw26\" (UID: \"c314533e-8a11-40dd-8549-77e5aa5e42a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26" Jul 07 18:31:36.879208 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:36.879174 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctkjf\" (UniqueName: \"kubernetes.io/projected/c314533e-8a11-40dd-8549-77e5aa5e42a7-kube-api-access-ctkjf\") pod \"kuadrant-operator-controller-manager-86878c769f-kkw26\" (UID: \"c314533e-8a11-40dd-8549-77e5aa5e42a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26" Jul 07 18:31:36.879208 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:36.879210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c314533e-8a11-40dd-8549-77e5aa5e42a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-kkw26\" (UID: \"c314533e-8a11-40dd-8549-77e5aa5e42a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26" Jul 07 18:31:36.879574 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:36.879553 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c314533e-8a11-40dd-8549-77e5aa5e42a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-kkw26\" (UID: \"c314533e-8a11-40dd-8549-77e5aa5e42a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26" Jul 07 18:31:36.892556 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:36.892525 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctkjf\" (UniqueName: \"kubernetes.io/projected/c314533e-8a11-40dd-8549-77e5aa5e42a7-kube-api-access-ctkjf\") pod \"kuadrant-operator-controller-manager-86878c769f-kkw26\" (UID: \"c314533e-8a11-40dd-8549-77e5aa5e42a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26" Jul 07 18:31:36.971920 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:36.971886 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26" Jul 07 18:31:37.103771 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:37.103749 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26"] Jul 07 18:31:37.106254 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:31:37.106227 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc314533e_8a11_40dd_8549_77e5aa5e42a7.slice/crio-fbd0a87f097dd3241f0affe0225b631bc22417d3d519b525202bcf395c735582 WatchSource:0}: Error finding container fbd0a87f097dd3241f0affe0225b631bc22417d3d519b525202bcf395c735582: Status 404 returned error can't find the container with id fbd0a87f097dd3241f0affe0225b631bc22417d3d519b525202bcf395c735582 Jul 07 18:31:37.160751 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:37.160723 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26" event={"ID":"c314533e-8a11-40dd-8549-77e5aa5e42a7","Type":"ContainerStarted","Data":"fbd0a87f097dd3241f0affe0225b631bc22417d3d519b525202bcf395c735582"} Jul 07 18:31:38.166668 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:38.166629 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26" event={"ID":"c314533e-8a11-40dd-8549-77e5aa5e42a7","Type":"ContainerStarted","Data":"6cf8c1e905f583d71468dc0380b8c680db8d083b6c7bd4e31659870855138aa7"} Jul 07 18:31:38.167078 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:38.166811 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26" Jul 07 18:31:38.190173 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:38.190124 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26" podStartSLOduration=2.190111471 podStartE2EDuration="2.190111471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:31:38.188678911 +0000 UTC m=+521.572472306" watchObservedRunningTime="2026-07-07 18:31:38.190111471 +0000 UTC m=+521.573904866" Jul 07 18:31:39.015151 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:39.015113 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:39.113165 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:39.113135 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:42.817921 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:42.817894 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:49.173551 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:49.173468 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26" Jul 07 18:31:49.238177 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:49.238146 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz"] Jul 07 18:31:49.238359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:49.238335 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz" podUID="99bdfb3a-2c41-47cc-825b-4209cb5230c5" containerName="manager" containerID="cri-o://f1d75bdf49b84ee1c3477ffee6574c4f359ed278c809475207fc2f3b87a62695" gracePeriod=10 Jul 07 18:31:49.481885 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:49.481860 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz" Jul 07 18:31:49.579790 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:49.579763 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gs8mx\" (UniqueName: \"kubernetes.io/projected/99bdfb3a-2c41-47cc-825b-4209cb5230c5-kube-api-access-gs8mx\") pod \"99bdfb3a-2c41-47cc-825b-4209cb5230c5\" (UID: \"99bdfb3a-2c41-47cc-825b-4209cb5230c5\") " Jul 07 18:31:49.579939 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:49.579822 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/99bdfb3a-2c41-47cc-825b-4209cb5230c5-extensions-socket-volume\") pod \"99bdfb3a-2c41-47cc-825b-4209cb5230c5\" (UID: \"99bdfb3a-2c41-47cc-825b-4209cb5230c5\") " Jul 07 18:31:49.580221 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:49.580200 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99bdfb3a-2c41-47cc-825b-4209cb5230c5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "99bdfb3a-2c41-47cc-825b-4209cb5230c5" (UID: "99bdfb3a-2c41-47cc-825b-4209cb5230c5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:31:49.581824 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:49.581803 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99bdfb3a-2c41-47cc-825b-4209cb5230c5-kube-api-access-gs8mx" (OuterVolumeSpecName: "kube-api-access-gs8mx") pod "99bdfb3a-2c41-47cc-825b-4209cb5230c5" (UID: "99bdfb3a-2c41-47cc-825b-4209cb5230c5"). InnerVolumeSpecName "kube-api-access-gs8mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:31:49.680973 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:49.680948 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/99bdfb3a-2c41-47cc-825b-4209cb5230c5-extensions-socket-volume\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:31:49.680973 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:49.680968 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs8mx\" (UniqueName: \"kubernetes.io/projected/99bdfb3a-2c41-47cc-825b-4209cb5230c5-kube-api-access-gs8mx\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:31:50.222374 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:50.222341 2578 generic.go:356] "Generic (PLEG): container finished" podID="99bdfb3a-2c41-47cc-825b-4209cb5230c5" containerID="f1d75bdf49b84ee1c3477ffee6574c4f359ed278c809475207fc2f3b87a62695" exitCode=0 Jul 07 18:31:50.222744 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:50.222401 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz" Jul 07 18:31:50.222744 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:50.222426 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz" event={"ID":"99bdfb3a-2c41-47cc-825b-4209cb5230c5","Type":"ContainerDied","Data":"f1d75bdf49b84ee1c3477ffee6574c4f359ed278c809475207fc2f3b87a62695"} Jul 07 18:31:50.222744 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:50.222481 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz" event={"ID":"99bdfb3a-2c41-47cc-825b-4209cb5230c5","Type":"ContainerDied","Data":"9b69c1a9b736cb83edace16a39b7c70aa7e4ef7b91da77b5879635d6750211ce"} Jul 07 18:31:50.222744 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:50.222505 2578 scope.go:117] "RemoveContainer" containerID="f1d75bdf49b84ee1c3477ffee6574c4f359ed278c809475207fc2f3b87a62695" Jul 07 18:31:50.232285 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:50.232266 2578 scope.go:117] "RemoveContainer" containerID="f1d75bdf49b84ee1c3477ffee6574c4f359ed278c809475207fc2f3b87a62695" Jul 07 18:31:50.232582 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:31:50.232562 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d75bdf49b84ee1c3477ffee6574c4f359ed278c809475207fc2f3b87a62695\": container with ID starting with f1d75bdf49b84ee1c3477ffee6574c4f359ed278c809475207fc2f3b87a62695 not found: ID does not exist" containerID="f1d75bdf49b84ee1c3477ffee6574c4f359ed278c809475207fc2f3b87a62695" Jul 07 18:31:50.232654 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:50.232588 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d75bdf49b84ee1c3477ffee6574c4f359ed278c809475207fc2f3b87a62695"} err="failed to get container status \"f1d75bdf49b84ee1c3477ffee6574c4f359ed278c809475207fc2f3b87a62695\": rpc error: code = NotFound desc = could not find container \"f1d75bdf49b84ee1c3477ffee6574c4f359ed278c809475207fc2f3b87a62695\": container with ID starting with f1d75bdf49b84ee1c3477ffee6574c4f359ed278c809475207fc2f3b87a62695 not found: ID does not exist" Jul 07 18:31:50.244035 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:50.244009 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz"] Jul 07 18:31:50.250581 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:50.250559 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s6fpz"] Jul 07 18:31:51.197718 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:51.197692 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99bdfb3a-2c41-47cc-825b-4209cb5230c5" path="/var/lib/kubelet/pods/99bdfb3a-2c41-47cc-825b-4209cb5230c5/volumes" Jul 07 18:31:52.190778 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.190737 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:52.289036 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.289003 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:52.694217 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.694185 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5"] Jul 07 18:31:52.694494 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.694482 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99bdfb3a-2c41-47cc-825b-4209cb5230c5" containerName="manager" Jul 07 18:31:52.694547 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.694495 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99bdfb3a-2c41-47cc-825b-4209cb5230c5" containerName="manager" Jul 07 18:31:52.694588 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.694550 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="99bdfb3a-2c41-47cc-825b-4209cb5230c5" containerName="manager" Jul 07 18:31:52.697394 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.697373 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.700040 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.700015 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 18:31:52.700040 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.700034 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9d2lf\"" Jul 07 18:31:52.710987 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.710968 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5"] Jul 07 18:31:52.724708 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.724680 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:52.800613 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.800586 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.800719 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.800616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.800719 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.800671 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.800848 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.800736 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.800848 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.800789 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.800848 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.800816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.800848 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.800840 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlm4w\" (UniqueName: \"kubernetes.io/projected/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-kube-api-access-wlm4w\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.801057 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.800947 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.801057 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.800976 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.902233 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.902211 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.902372 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.902239 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.902372 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.902257 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlm4w\" (UniqueName: \"kubernetes.io/projected/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-kube-api-access-wlm4w\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.902372 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.902276 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.902372 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.902299 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.902372 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.902355 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.902625 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.902387 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.902625 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.902460 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.902625 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.902492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.902770 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.902678 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.902843 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.902790 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.902923 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.902899 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.902980 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.902901 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.903197 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.903182 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.904558 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.904537 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.904755 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.904740 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.905493 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.905442 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:31:52.911171 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.911153 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:52.911444 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:52.911423 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlm4w\" (UniqueName: \"kubernetes.io/projected/b39b4b49-06c6-4309-ac4e-ca8c140a9a2d-kube-api-access-wlm4w\") pod \"maas-default-gateway-openshift-default-56654dcf7-65gv5\" (UID: \"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:53.007725 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:53.007702 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:53.135985 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:53.135961 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5"] Jul 07 18:31:53.138394 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:31:53.138370 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39b4b49_06c6_4309_ac4e_ca8c140a9a2d.slice/crio-b132c2c15c4dbf45ee910268d8ce9d3228e633fb3d44bfbe3eb8ad07fcb995e3 WatchSource:0}: Error finding container b132c2c15c4dbf45ee910268d8ce9d3228e633fb3d44bfbe3eb8ad07fcb995e3: Status 404 returned error can't find the container with id b132c2c15c4dbf45ee910268d8ce9d3228e633fb3d44bfbe3eb8ad07fcb995e3 Jul 07 18:31:53.235573 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:53.235545 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" event={"ID":"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d","Type":"ContainerStarted","Data":"b132c2c15c4dbf45ee910268d8ce9d3228e633fb3d44bfbe3eb8ad07fcb995e3"} Jul 07 18:31:55.971181 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:55.971144 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:31:55.971485 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:55.971216 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:31:55.971485 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:55.971249 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:31:56.251429 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:56.251348 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" event={"ID":"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d","Type":"ContainerStarted","Data":"40c05868cbde9d3dccbd00d60ca5656901c71f94997eef81b8c23efa6652d1b4"} Jul 07 18:31:56.275693 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:56.275636 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" podStartSLOduration=1.44482876 podStartE2EDuration="4.275590178s" totalImagesPullingTime="2.830761418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:31:56.27479098 +0000 UTC m=+539.658584401" watchObservedRunningTime="2026-07-07 18:31:56.275590178 +0000 UTC m=+539.659383585" Jul 07 18:31:57.008338 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:57.008297 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:57.013342 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:57.013290 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:57.255163 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:57.255138 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:31:57.255964 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:31:57.255946 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:32:07.038910 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.038880 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:32:07.164624 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.164590 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:32:07.230054 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.230024 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:32:07.513399 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.513374 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:32:07.516379 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.516358 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-74l2w" Jul 07 18:32:07.518903 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.518878 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 07 18:32:07.519000 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.518947 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-z7bnh\"" Jul 07 18:32:07.527245 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.527228 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:32:07.569780 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.569756 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:32:07.609409 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.609385 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8d18b117-f206-4c2c-a9db-432663ab515d-config-file\") pod \"limitador-limitador-54d6d988d5-74l2w\" (UID: \"8d18b117-f206-4c2c-a9db-432663ab515d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-74l2w" Jul 07 18:32:07.609496 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.609422 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx4fw\" (UniqueName: \"kubernetes.io/projected/8d18b117-f206-4c2c-a9db-432663ab515d-kube-api-access-lx4fw\") pod \"limitador-limitador-54d6d988d5-74l2w\" (UID: \"8d18b117-f206-4c2c-a9db-432663ab515d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-74l2w" Jul 07 18:32:07.710561 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.710540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8d18b117-f206-4c2c-a9db-432663ab515d-config-file\") pod \"limitador-limitador-54d6d988d5-74l2w\" (UID: \"8d18b117-f206-4c2c-a9db-432663ab515d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-74l2w" Jul 07 18:32:07.710648 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.710572 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx4fw\" (UniqueName: \"kubernetes.io/projected/8d18b117-f206-4c2c-a9db-432663ab515d-kube-api-access-lx4fw\") pod \"limitador-limitador-54d6d988d5-74l2w\" (UID: \"8d18b117-f206-4c2c-a9db-432663ab515d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-74l2w" Jul 07 18:32:07.711106 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.711088 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8d18b117-f206-4c2c-a9db-432663ab515d-config-file\") pod \"limitador-limitador-54d6d988d5-74l2w\" (UID: \"8d18b117-f206-4c2c-a9db-432663ab515d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-74l2w" Jul 07 18:32:07.718900 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.718872 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx4fw\" (UniqueName: \"kubernetes.io/projected/8d18b117-f206-4c2c-a9db-432663ab515d-kube-api-access-lx4fw\") pod \"limitador-limitador-54d6d988d5-74l2w\" (UID: \"8d18b117-f206-4c2c-a9db-432663ab515d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-74l2w" Jul 07 18:32:07.826981 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.826926 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-74l2w" Jul 07 18:32:07.932559 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.932534 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-qv7wx"] Jul 07 18:32:07.938893 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.938868 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qv7wx" Jul 07 18:32:07.953458 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.953439 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-p868l\"" Jul 07 18:32:07.967614 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.967595 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qv7wx"] Jul 07 18:32:07.980210 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:07.980191 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:32:08.006289 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:32:08.006268 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d18b117_f206_4c2c_a9db_432663ab515d.slice/crio-6b8cbb1a91dd320195353192396a8a396a73f16a1aeddb2a07781f2dda339aa5 WatchSource:0}: Error finding container 6b8cbb1a91dd320195353192396a8a396a73f16a1aeddb2a07781f2dda339aa5: Status 404 returned error can't find the container with id 6b8cbb1a91dd320195353192396a8a396a73f16a1aeddb2a07781f2dda339aa5 Jul 07 18:32:08.012667 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:08.012649 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74vrv\" (UniqueName: \"kubernetes.io/projected/4c179b3c-a35f-455e-9c08-bd3a0547ca0e-kube-api-access-74vrv\") pod \"authorino-58cd955675-qv7wx\" (UID: \"4c179b3c-a35f-455e-9c08-bd3a0547ca0e\") " pod="kuadrant-system/authorino-58cd955675-qv7wx" Jul 07 18:32:08.044391 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:08.044372 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:32:08.113039 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:08.112992 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74vrv\" (UniqueName: \"kubernetes.io/projected/4c179b3c-a35f-455e-9c08-bd3a0547ca0e-kube-api-access-74vrv\") pod \"authorino-58cd955675-qv7wx\" (UID: \"4c179b3c-a35f-455e-9c08-bd3a0547ca0e\") " pod="kuadrant-system/authorino-58cd955675-qv7wx" Jul 07 18:32:08.124724 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:08.124701 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74vrv\" (UniqueName: \"kubernetes.io/projected/4c179b3c-a35f-455e-9c08-bd3a0547ca0e-kube-api-access-74vrv\") pod \"authorino-58cd955675-qv7wx\" (UID: \"4c179b3c-a35f-455e-9c08-bd3a0547ca0e\") " pod="kuadrant-system/authorino-58cd955675-qv7wx" Jul 07 18:32:08.143787 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:08.143767 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:32:08.248113 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:08.248090 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qv7wx" Jul 07 18:32:08.303458 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:08.303425 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-74l2w" event={"ID":"8d18b117-f206-4c2c-a9db-432663ab515d","Type":"ContainerStarted","Data":"6b8cbb1a91dd320195353192396a8a396a73f16a1aeddb2a07781f2dda339aa5"} Jul 07 18:32:08.376363 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:08.376273 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qv7wx"] Jul 07 18:32:08.379438 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:32:08.379414 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c179b3c_a35f_455e_9c08_bd3a0547ca0e.slice/crio-a7add0daaa58212bd78d05aad6cb74c17b4aa80c00f0db34ddbcdc42dd85427d WatchSource:0}: Error finding container a7add0daaa58212bd78d05aad6cb74c17b4aa80c00f0db34ddbcdc42dd85427d: Status 404 returned error can't find the container with id a7add0daaa58212bd78d05aad6cb74c17b4aa80c00f0db34ddbcdc42dd85427d Jul 07 18:32:09.309827 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:09.309789 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qv7wx" event={"ID":"4c179b3c-a35f-455e-9c08-bd3a0547ca0e","Type":"ContainerStarted","Data":"a7add0daaa58212bd78d05aad6cb74c17b4aa80c00f0db34ddbcdc42dd85427d"} Jul 07 18:32:13.334014 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:13.333965 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-74l2w" event={"ID":"8d18b117-f206-4c2c-a9db-432663ab515d","Type":"ContainerStarted","Data":"e8d74a0045c9cca8921abba6f26d9be45dd7db0a19894ec4cb073b3b120785d6"} Jul 07 18:32:13.334490 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:13.334066 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-74l2w" Jul 07 18:32:13.335498 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:13.335476 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qv7wx" event={"ID":"4c179b3c-a35f-455e-9c08-bd3a0547ca0e","Type":"ContainerStarted","Data":"03b58f4d33fd7b8bc42824057c935bbc7700e2ce3d384c9379571f37efea5c70"} Jul 07 18:32:13.351158 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:13.351118 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-74l2w" podStartSLOduration=1.937575357 podStartE2EDuration="6.351107021s" totalImagesPullingTime="4.413531664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:32:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:32:13.350018138 +0000 UTC m=+556.733811533" watchObservedRunningTime="2026-07-07 18:32:13.351107021 +0000 UTC m=+556.734900414" Jul 07 18:32:13.364505 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:13.364469 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-qv7wx" podStartSLOduration=2.321814196 podStartE2EDuration="6.364457305s" totalImagesPullingTime="4.042643109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:32:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:32:13.363516296 +0000 UTC m=+556.747309689" watchObservedRunningTime="2026-07-07 18:32:13.364457305 +0000 UTC m=+556.748250697" Jul 07 18:32:13.408261 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:13.408233 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qv7wx"] Jul 07 18:32:15.343912 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:15.343878 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-qv7wx" podUID="4c179b3c-a35f-455e-9c08-bd3a0547ca0e" containerName="authorino" containerID="cri-o://03b58f4d33fd7b8bc42824057c935bbc7700e2ce3d384c9379571f37efea5c70" gracePeriod=30 Jul 07 18:32:15.577291 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:15.577269 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qv7wx" Jul 07 18:32:15.673846 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:15.673788 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74vrv\" (UniqueName: \"kubernetes.io/projected/4c179b3c-a35f-455e-9c08-bd3a0547ca0e-kube-api-access-74vrv\") pod \"4c179b3c-a35f-455e-9c08-bd3a0547ca0e\" (UID: \"4c179b3c-a35f-455e-9c08-bd3a0547ca0e\") " Jul 07 18:32:15.675651 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:15.675628 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c179b3c-a35f-455e-9c08-bd3a0547ca0e-kube-api-access-74vrv" (OuterVolumeSpecName: "kube-api-access-74vrv") pod "4c179b3c-a35f-455e-9c08-bd3a0547ca0e" (UID: "4c179b3c-a35f-455e-9c08-bd3a0547ca0e"). InnerVolumeSpecName "kube-api-access-74vrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:32:15.774647 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:15.774621 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74vrv\" (UniqueName: \"kubernetes.io/projected/4c179b3c-a35f-455e-9c08-bd3a0547ca0e-kube-api-access-74vrv\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:32:16.349432 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:16.349408 2578 generic.go:356] "Generic (PLEG): container finished" podID="4c179b3c-a35f-455e-9c08-bd3a0547ca0e" containerID="03b58f4d33fd7b8bc42824057c935bbc7700e2ce3d384c9379571f37efea5c70" exitCode=0 Jul 07 18:32:16.349757 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:16.349449 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qv7wx" Jul 07 18:32:16.349757 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:16.349479 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qv7wx" event={"ID":"4c179b3c-a35f-455e-9c08-bd3a0547ca0e","Type":"ContainerDied","Data":"03b58f4d33fd7b8bc42824057c935bbc7700e2ce3d384c9379571f37efea5c70"} Jul 07 18:32:16.349757 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:16.349507 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qv7wx" event={"ID":"4c179b3c-a35f-455e-9c08-bd3a0547ca0e","Type":"ContainerDied","Data":"a7add0daaa58212bd78d05aad6cb74c17b4aa80c00f0db34ddbcdc42dd85427d"} Jul 07 18:32:16.349757 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:16.349524 2578 scope.go:117] "RemoveContainer" containerID="03b58f4d33fd7b8bc42824057c935bbc7700e2ce3d384c9379571f37efea5c70" Jul 07 18:32:16.357682 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:16.357665 2578 scope.go:117] "RemoveContainer" containerID="03b58f4d33fd7b8bc42824057c935bbc7700e2ce3d384c9379571f37efea5c70" Jul 07 18:32:16.357952 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:32:16.357931 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03b58f4d33fd7b8bc42824057c935bbc7700e2ce3d384c9379571f37efea5c70\": container with ID starting with 03b58f4d33fd7b8bc42824057c935bbc7700e2ce3d384c9379571f37efea5c70 not found: ID does not exist" containerID="03b58f4d33fd7b8bc42824057c935bbc7700e2ce3d384c9379571f37efea5c70" Jul 07 18:32:16.358017 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:16.357965 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b58f4d33fd7b8bc42824057c935bbc7700e2ce3d384c9379571f37efea5c70"} err="failed to get container status \"03b58f4d33fd7b8bc42824057c935bbc7700e2ce3d384c9379571f37efea5c70\": rpc error: code = NotFound desc = could not find container \"03b58f4d33fd7b8bc42824057c935bbc7700e2ce3d384c9379571f37efea5c70\": container with ID starting with 03b58f4d33fd7b8bc42824057c935bbc7700e2ce3d384c9379571f37efea5c70 not found: ID does not exist" Jul 07 18:32:16.369498 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:16.369475 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qv7wx"] Jul 07 18:32:16.373283 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:16.373263 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-qv7wx"] Jul 07 18:32:17.196847 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:17.196824 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c179b3c-a35f-455e-9c08-bd3a0547ca0e" path="/var/lib/kubelet/pods/4c179b3c-a35f-455e-9c08-bd3a0547ca0e/volumes" Jul 07 18:32:24.340087 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:24.340059 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-74l2w" Jul 07 18:32:29.451268 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:29.451236 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:32:45.625742 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:45.625704 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:32:45.876194 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:45.876117 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bbzrb"] Jul 07 18:32:45.876451 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:45.876438 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c179b3c-a35f-455e-9c08-bd3a0547ca0e" containerName="authorino" Jul 07 18:32:45.876498 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:45.876453 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c179b3c-a35f-455e-9c08-bd3a0547ca0e" containerName="authorino" Jul 07 18:32:45.876544 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:45.876517 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c179b3c-a35f-455e-9c08-bd3a0547ca0e" containerName="authorino" Jul 07 18:32:45.949840 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:45.949809 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:32:45.970205 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:45.970180 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bbzrb"] Jul 07 18:32:45.970345 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:45.970263 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bbzrb" Jul 07 18:32:45.972762 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:45.972742 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-p868l\"" Jul 07 18:32:46.075560 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.075537 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pngn\" (UniqueName: \"kubernetes.io/projected/68838979-bee6-42d8-a948-80e0362ff823-kube-api-access-6pngn\") pod \"authorino-5d4b67c446-bbzrb\" (UID: \"68838979-bee6-42d8-a948-80e0362ff823\") " pod="kuadrant-system/authorino-5d4b67c446-bbzrb" Jul 07 18:32:46.124694 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.124675 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bbzrb"] Jul 07 18:32:46.124879 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:32:46.124854 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6pngn], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-bbzrb" podUID="68838979-bee6-42d8-a948-80e0362ff823" Jul 07 18:32:46.149649 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.149601 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-758bb4d4c9-phcgr"] Jul 07 18:32:46.173784 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.173753 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-758bb4d4c9-phcgr"] Jul 07 18:32:46.173784 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.173769 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-758bb4d4c9-phcgr" Jul 07 18:32:46.176093 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.176076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pngn\" (UniqueName: \"kubernetes.io/projected/68838979-bee6-42d8-a948-80e0362ff823-kube-api-access-6pngn\") pod \"authorino-5d4b67c446-bbzrb\" (UID: \"68838979-bee6-42d8-a948-80e0362ff823\") " pod="kuadrant-system/authorino-5d4b67c446-bbzrb" Jul 07 18:32:46.184743 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.184718 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pngn\" (UniqueName: \"kubernetes.io/projected/68838979-bee6-42d8-a948-80e0362ff823-kube-api-access-6pngn\") pod \"authorino-5d4b67c446-bbzrb\" (UID: \"68838979-bee6-42d8-a948-80e0362ff823\") " pod="kuadrant-system/authorino-5d4b67c446-bbzrb" Jul 07 18:32:46.232827 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.232804 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-758bb4d4c9-phcgr"] Jul 07 18:32:46.232975 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:32:46.232960 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-7nr9q], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-758bb4d4c9-phcgr" podUID="8146ef15-b45f-47e7-837e-60209d171fb7" Jul 07 18:32:46.265108 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.265085 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-796cb9b767-vmhxm"] Jul 07 18:32:46.276419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.276400 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nr9q\" (UniqueName: \"kubernetes.io/projected/8146ef15-b45f-47e7-837e-60209d171fb7-kube-api-access-7nr9q\") pod \"authorino-758bb4d4c9-phcgr\" (UID: \"8146ef15-b45f-47e7-837e-60209d171fb7\") " pod="kuadrant-system/authorino-758bb4d4c9-phcgr" Jul 07 18:32:46.285864 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.285843 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-796cb9b767-vmhxm"] Jul 07 18:32:46.285948 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.285941 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-796cb9b767-vmhxm" Jul 07 18:32:46.288226 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.288210 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 07 18:32:46.377550 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.377532 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg5zg\" (UniqueName: \"kubernetes.io/projected/724484d5-0bd8-44fa-abed-c67bbb1bd9a8-kube-api-access-mg5zg\") pod \"authorino-796cb9b767-vmhxm\" (UID: \"724484d5-0bd8-44fa-abed-c67bbb1bd9a8\") " pod="kuadrant-system/authorino-796cb9b767-vmhxm" Jul 07 18:32:46.377654 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.377565 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/724484d5-0bd8-44fa-abed-c67bbb1bd9a8-tls-cert\") pod \"authorino-796cb9b767-vmhxm\" (UID: \"724484d5-0bd8-44fa-abed-c67bbb1bd9a8\") " pod="kuadrant-system/authorino-796cb9b767-vmhxm" Jul 07 18:32:46.377654 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.377590 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nr9q\" (UniqueName: \"kubernetes.io/projected/8146ef15-b45f-47e7-837e-60209d171fb7-kube-api-access-7nr9q\") pod \"authorino-758bb4d4c9-phcgr\" (UID: \"8146ef15-b45f-47e7-837e-60209d171fb7\") " pod="kuadrant-system/authorino-758bb4d4c9-phcgr" Jul 07 18:32:46.385763 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.385735 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nr9q\" (UniqueName: \"kubernetes.io/projected/8146ef15-b45f-47e7-837e-60209d171fb7-kube-api-access-7nr9q\") pod \"authorino-758bb4d4c9-phcgr\" (UID: \"8146ef15-b45f-47e7-837e-60209d171fb7\") " pod="kuadrant-system/authorino-758bb4d4c9-phcgr" Jul 07 18:32:46.467985 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.467924 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-758bb4d4c9-phcgr" Jul 07 18:32:46.468103 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.467924 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bbzrb" Jul 07 18:32:46.478263 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.478242 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg5zg\" (UniqueName: \"kubernetes.io/projected/724484d5-0bd8-44fa-abed-c67bbb1bd9a8-kube-api-access-mg5zg\") pod \"authorino-796cb9b767-vmhxm\" (UID: \"724484d5-0bd8-44fa-abed-c67bbb1bd9a8\") " pod="kuadrant-system/authorino-796cb9b767-vmhxm" Jul 07 18:32:46.478382 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.478278 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/724484d5-0bd8-44fa-abed-c67bbb1bd9a8-tls-cert\") pod \"authorino-796cb9b767-vmhxm\" (UID: \"724484d5-0bd8-44fa-abed-c67bbb1bd9a8\") " pod="kuadrant-system/authorino-796cb9b767-vmhxm" Jul 07 18:32:46.480348 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.480311 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/724484d5-0bd8-44fa-abed-c67bbb1bd9a8-tls-cert\") pod \"authorino-796cb9b767-vmhxm\" (UID: \"724484d5-0bd8-44fa-abed-c67bbb1bd9a8\") " pod="kuadrant-system/authorino-796cb9b767-vmhxm" Jul 07 18:32:46.486062 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.486042 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg5zg\" (UniqueName: \"kubernetes.io/projected/724484d5-0bd8-44fa-abed-c67bbb1bd9a8-kube-api-access-mg5zg\") pod \"authorino-796cb9b767-vmhxm\" (UID: \"724484d5-0bd8-44fa-abed-c67bbb1bd9a8\") " pod="kuadrant-system/authorino-796cb9b767-vmhxm" Jul 07 18:32:46.493353 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.493335 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-758bb4d4c9-phcgr" Jul 07 18:32:46.510932 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.510916 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bbzrb" Jul 07 18:32:46.579192 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.579168 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nr9q\" (UniqueName: \"kubernetes.io/projected/8146ef15-b45f-47e7-837e-60209d171fb7-kube-api-access-7nr9q\") pod \"8146ef15-b45f-47e7-837e-60209d171fb7\" (UID: \"8146ef15-b45f-47e7-837e-60209d171fb7\") " Jul 07 18:32:46.580907 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.580888 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8146ef15-b45f-47e7-837e-60209d171fb7-kube-api-access-7nr9q" (OuterVolumeSpecName: "kube-api-access-7nr9q") pod "8146ef15-b45f-47e7-837e-60209d171fb7" (UID: "8146ef15-b45f-47e7-837e-60209d171fb7"). InnerVolumeSpecName "kube-api-access-7nr9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:32:46.595020 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.595002 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-796cb9b767-vmhxm" Jul 07 18:32:46.680108 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.680083 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pngn\" (UniqueName: \"kubernetes.io/projected/68838979-bee6-42d8-a948-80e0362ff823-kube-api-access-6pngn\") pod \"68838979-bee6-42d8-a948-80e0362ff823\" (UID: \"68838979-bee6-42d8-a948-80e0362ff823\") " Jul 07 18:32:46.680483 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.680405 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nr9q\" (UniqueName: \"kubernetes.io/projected/8146ef15-b45f-47e7-837e-60209d171fb7-kube-api-access-7nr9q\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:32:46.682066 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.682040 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68838979-bee6-42d8-a948-80e0362ff823-kube-api-access-6pngn" (OuterVolumeSpecName: "kube-api-access-6pngn") pod "68838979-bee6-42d8-a948-80e0362ff823" (UID: "68838979-bee6-42d8-a948-80e0362ff823"). InnerVolumeSpecName "kube-api-access-6pngn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:32:46.745133 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.745112 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-796cb9b767-vmhxm"] Jul 07 18:32:46.747225 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:32:46.747201 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod724484d5_0bd8_44fa_abed_c67bbb1bd9a8.slice/crio-c0e81533467ffec83d168236105f6555ac73071fec79c594312c033445e62ec1 WatchSource:0}: Error finding container c0e81533467ffec83d168236105f6555ac73071fec79c594312c033445e62ec1: Status 404 returned error can't find the container with id c0e81533467ffec83d168236105f6555ac73071fec79c594312c033445e62ec1 Jul 07 18:32:46.781716 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:46.781695 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pngn\" (UniqueName: \"kubernetes.io/projected/68838979-bee6-42d8-a948-80e0362ff823-kube-api-access-6pngn\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:32:47.473290 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:47.473247 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-796cb9b767-vmhxm" event={"ID":"724484d5-0bd8-44fa-abed-c67bbb1bd9a8","Type":"ContainerStarted","Data":"a593a57324784ea9bfeba987c4f63a7ff4d9f0bcc563fd77b7ae18b7a4411071"} Jul 07 18:32:47.473290 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:47.473288 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bbzrb" Jul 07 18:32:47.473543 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:47.473302 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-796cb9b767-vmhxm" event={"ID":"724484d5-0bd8-44fa-abed-c67bbb1bd9a8","Type":"ContainerStarted","Data":"c0e81533467ffec83d168236105f6555ac73071fec79c594312c033445e62ec1"} Jul 07 18:32:47.473543 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:47.473264 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-758bb4d4c9-phcgr" Jul 07 18:32:47.490006 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:47.489957 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-796cb9b767-vmhxm" podStartSLOduration=1.075194803 podStartE2EDuration="1.489946169s" totalImagesPullingTime="414.751366ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:32:47.488570213 +0000 UTC m=+590.872363617" watchObservedRunningTime="2026-07-07 18:32:47.489946169 +0000 UTC m=+590.873739563" Jul 07 18:32:47.514074 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:47.514040 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bbzrb"] Jul 07 18:32:47.524128 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:47.524107 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bbzrb"] Jul 07 18:32:47.554919 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:47.554893 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-758bb4d4c9-phcgr"] Jul 07 18:32:47.563150 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:47.563129 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-758bb4d4c9-phcgr"] Jul 07 18:32:49.196454 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:49.196426 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68838979-bee6-42d8-a948-80e0362ff823" path="/var/lib/kubelet/pods/68838979-bee6-42d8-a948-80e0362ff823/volumes" Jul 07 18:32:49.196793 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:49.196630 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8146ef15-b45f-47e7-837e-60209d171fb7" path="/var/lib/kubelet/pods/8146ef15-b45f-47e7-837e-60209d171fb7/volumes" Jul 07 18:32:52.313610 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:52.313566 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:32:52.430885 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:52.430860 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:32:52.532460 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:52.532432 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:32:55.515533 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:55.515461 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:32:55.615179 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:55.615150 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:32:57.057573 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:57.057549 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 18:32:57.057948 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:32:57.057799 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 18:33:28.025785 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:28.025752 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:33:28.563061 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:28.563025 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:33:28.779668 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:28.779632 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:33:29.273562 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:29.273497 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:33:29.436848 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:29.436811 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:33:29.552111 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:29.552031 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:33:36.685330 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:36.685293 2578 generic.go:356] "Generic (PLEG): container finished" podID="b39b4b49-06c6-4309-ac4e-ca8c140a9a2d" containerID="40c05868cbde9d3dccbd00d60ca5656901c71f94997eef81b8c23efa6652d1b4" exitCode=0 Jul 07 18:33:36.685768 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:36.685371 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" event={"ID":"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d","Type":"ContainerDied","Data":"40c05868cbde9d3dccbd00d60ca5656901c71f94997eef81b8c23efa6652d1b4"} Jul 07 18:33:36.685857 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:36.685841 2578 scope.go:117] "RemoveContainer" containerID="40c05868cbde9d3dccbd00d60ca5656901c71f94997eef81b8c23efa6652d1b4" Jul 07 18:33:36.686793 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:36.686768 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:33:36.686851 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:36.686833 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:33:36.686896 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:36.686867 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:33:37.695021 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:37.694928 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" event={"ID":"b39b4b49-06c6-4309-ac4e-ca8c140a9a2d","Type":"ContainerStarted","Data":"70c4f81f242a5e2ddc10b5b4a35f4345fad2e63a6900f43d0d3daf4b6aabec3c"} Jul 07 18:33:38.008126 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:38.008090 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:33:38.009288 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:38.009261 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-65gv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 07 18:33:38.009424 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:38.009311 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" podUID="b39b4b49-06c6-4309-ac4e-ca8c140a9a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 07 18:33:38.777502 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:38.777462 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:33:38.879447 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:38.879416 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:33:39.008573 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:39.008539 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-65gv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 07 18:33:39.008733 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:39.008606 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" podUID="b39b4b49-06c6-4309-ac4e-ca8c140a9a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 07 18:33:40.008516 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:40.008477 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-65gv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 07 18:33:40.008961 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:40.008535 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" podUID="b39b4b49-06c6-4309-ac4e-ca8c140a9a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 07 18:33:41.008790 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:41.008756 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-65gv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 07 18:33:41.009238 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:41.008820 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" podUID="b39b4b49-06c6-4309-ac4e-ca8c140a9a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 07 18:33:42.011861 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:42.011832 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:33:42.012212 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:42.012091 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:33:42.012689 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:42.012674 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-65gv5" Jul 07 18:33:55.271081 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:55.271043 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:33:55.632034 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:33:55.631944 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:34:07.376430 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:07.376381 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:34:08.050490 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:08.050439 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:34:10.385610 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:10.385563 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:34:10.486909 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:10.486873 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:34:12.898521 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:12.898475 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:34:13.943760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:13.943726 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:34:19.564815 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:19.564781 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:34:20.945092 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:20.945055 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:34:23.428747 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.428715 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87"] Jul 07 18:34:23.431463 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.431442 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:23.434021 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.434002 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 18:34:23.435174 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.435102 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 18:34:23.435174 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.435157 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8spls\"" Jul 07 18:34:23.435359 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.435157 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 07 18:34:23.444017 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.443997 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87"] Jul 07 18:34:23.472213 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.472183 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:34:23.519730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.519704 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df3f97a-0327-48e6-862d-8bc4d58fc99c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qwn87\" (UID: \"7df3f97a-0327-48e6-862d-8bc4d58fc99c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:23.519830 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.519732 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7df3f97a-0327-48e6-862d-8bc4d58fc99c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qwn87\" (UID: \"7df3f97a-0327-48e6-862d-8bc4d58fc99c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:23.519830 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.519769 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh4kk\" (UniqueName: \"kubernetes.io/projected/7df3f97a-0327-48e6-862d-8bc4d58fc99c-kube-api-access-nh4kk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qwn87\" (UID: \"7df3f97a-0327-48e6-862d-8bc4d58fc99c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:23.519923 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.519841 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7df3f97a-0327-48e6-862d-8bc4d58fc99c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qwn87\" (UID: \"7df3f97a-0327-48e6-862d-8bc4d58fc99c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:23.519923 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.519859 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7df3f97a-0327-48e6-862d-8bc4d58fc99c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qwn87\" (UID: \"7df3f97a-0327-48e6-862d-8bc4d58fc99c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:23.519923 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.519876 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7df3f97a-0327-48e6-862d-8bc4d58fc99c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qwn87\" (UID: \"7df3f97a-0327-48e6-862d-8bc4d58fc99c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:23.621030 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.621001 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7df3f97a-0327-48e6-862d-8bc4d58fc99c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qwn87\" (UID: \"7df3f97a-0327-48e6-862d-8bc4d58fc99c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:23.621030 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.621032 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7df3f97a-0327-48e6-862d-8bc4d58fc99c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qwn87\" (UID: \"7df3f97a-0327-48e6-862d-8bc4d58fc99c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:23.621220 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.621069 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df3f97a-0327-48e6-862d-8bc4d58fc99c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qwn87\" (UID: \"7df3f97a-0327-48e6-862d-8bc4d58fc99c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:23.621220 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.621087 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7df3f97a-0327-48e6-862d-8bc4d58fc99c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qwn87\" (UID: \"7df3f97a-0327-48e6-862d-8bc4d58fc99c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:23.621220 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.621118 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh4kk\" (UniqueName: \"kubernetes.io/projected/7df3f97a-0327-48e6-862d-8bc4d58fc99c-kube-api-access-nh4kk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qwn87\" (UID: \"7df3f97a-0327-48e6-862d-8bc4d58fc99c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:23.621220 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.621147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7df3f97a-0327-48e6-862d-8bc4d58fc99c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qwn87\" (UID: \"7df3f97a-0327-48e6-862d-8bc4d58fc99c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:23.621502 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.621474 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df3f97a-0327-48e6-862d-8bc4d58fc99c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qwn87\" (UID: \"7df3f97a-0327-48e6-862d-8bc4d58fc99c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:23.621502 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.621489 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7df3f97a-0327-48e6-862d-8bc4d58fc99c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qwn87\" (UID: \"7df3f97a-0327-48e6-862d-8bc4d58fc99c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:23.621625 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.621535 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7df3f97a-0327-48e6-862d-8bc4d58fc99c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qwn87\" (UID: \"7df3f97a-0327-48e6-862d-8bc4d58fc99c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:23.623273 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.623250 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7df3f97a-0327-48e6-862d-8bc4d58fc99c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qwn87\" (UID: \"7df3f97a-0327-48e6-862d-8bc4d58fc99c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:23.623454 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.623439 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7df3f97a-0327-48e6-862d-8bc4d58fc99c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qwn87\" (UID: \"7df3f97a-0327-48e6-862d-8bc4d58fc99c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:23.629066 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.629044 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh4kk\" (UniqueName: \"kubernetes.io/projected/7df3f97a-0327-48e6-862d-8bc4d58fc99c-kube-api-access-nh4kk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qwn87\" (UID: \"7df3f97a-0327-48e6-862d-8bc4d58fc99c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:23.743718 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.743697 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:23.871642 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.871611 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87"] Jul 07 18:34:23.880718 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:34:23.880692 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df3f97a_0327_48e6_862d_8bc4d58fc99c.slice/crio-5318f372bf3589df7a5a76c1e276bdac299138d8d03a9260c3d128837e77e529 WatchSource:0}: Error finding container 5318f372bf3589df7a5a76c1e276bdac299138d8d03a9260c3d128837e77e529: Status 404 returned error can't find the container with id 5318f372bf3589df7a5a76c1e276bdac299138d8d03a9260c3d128837e77e529 Jul 07 18:34:23.882374 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.882357 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 18:34:23.903565 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:23.903539 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" event={"ID":"7df3f97a-0327-48e6-862d-8bc4d58fc99c","Type":"ContainerStarted","Data":"5318f372bf3589df7a5a76c1e276bdac299138d8d03a9260c3d128837e77e529"} Jul 07 18:34:25.913476 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:25.913442 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" event={"ID":"7df3f97a-0327-48e6-862d-8bc4d58fc99c","Type":"ContainerStarted","Data":"6e754c0a75a610dde645b17408f0ae03f75e9c14265efd81aa2f9c940800ccf8"} Jul 07 18:34:25.932510 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:25.932467 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" podStartSLOduration=1.892884214 podStartE2EDuration="2.932453456s" totalImagesPullingTime="1.039569242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:34:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:34:25.931966667 +0000 UTC m=+689.315760060" watchObservedRunningTime="2026-07-07 18:34:25.932453456 +0000 UTC m=+689.316246850" Jul 07 18:34:26.346339 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:26.346285 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:34:33.744692 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:33.744663 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:33.745033 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:33.744712 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:33.756968 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:33.756947 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:33.961512 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:33.961482 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qwn87" Jul 07 18:34:36.827613 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:36.827584 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7"] Jul 07 18:34:36.834076 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:36.834044 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:36.837472 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:36.837442 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 07 18:34:36.842121 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:36.842101 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7"] Jul 07 18:34:36.869072 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:36.869043 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:34:36.915732 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:36.915708 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/130c95ce-3eff-4c85-a5de-719730e28566-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7\" (UID: \"130c95ce-3eff-4c85-a5de-719730e28566\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:36.915829 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:36.915742 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/130c95ce-3eff-4c85-a5de-719730e28566-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7\" (UID: \"130c95ce-3eff-4c85-a5de-719730e28566\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:36.915829 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:36.915781 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/130c95ce-3eff-4c85-a5de-719730e28566-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7\" (UID: \"130c95ce-3eff-4c85-a5de-719730e28566\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:36.915913 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:36.915879 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/130c95ce-3eff-4c85-a5de-719730e28566-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7\" (UID: \"130c95ce-3eff-4c85-a5de-719730e28566\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:36.915966 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:36.915916 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/130c95ce-3eff-4c85-a5de-719730e28566-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7\" (UID: \"130c95ce-3eff-4c85-a5de-719730e28566\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:36.916002 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:36.915985 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8f8g\" (UniqueName: \"kubernetes.io/projected/130c95ce-3eff-4c85-a5de-719730e28566-kube-api-access-w8f8g\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7\" (UID: \"130c95ce-3eff-4c85-a5de-719730e28566\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:37.016478 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:37.016456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8f8g\" (UniqueName: \"kubernetes.io/projected/130c95ce-3eff-4c85-a5de-719730e28566-kube-api-access-w8f8g\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7\" (UID: \"130c95ce-3eff-4c85-a5de-719730e28566\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:37.016626 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:37.016486 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/130c95ce-3eff-4c85-a5de-719730e28566-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7\" (UID: \"130c95ce-3eff-4c85-a5de-719730e28566\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:37.016626 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:37.016511 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/130c95ce-3eff-4c85-a5de-719730e28566-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7\" (UID: \"130c95ce-3eff-4c85-a5de-719730e28566\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:37.016626 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:37.016558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/130c95ce-3eff-4c85-a5de-719730e28566-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7\" (UID: \"130c95ce-3eff-4c85-a5de-719730e28566\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:37.016626 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:37.016603 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/130c95ce-3eff-4c85-a5de-719730e28566-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7\" (UID: \"130c95ce-3eff-4c85-a5de-719730e28566\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:37.016825 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:37.016628 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/130c95ce-3eff-4c85-a5de-719730e28566-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7\" (UID: \"130c95ce-3eff-4c85-a5de-719730e28566\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:37.017041 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:37.017013 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/130c95ce-3eff-4c85-a5de-719730e28566-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7\" (UID: \"130c95ce-3eff-4c85-a5de-719730e28566\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:37.017041 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:37.017040 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/130c95ce-3eff-4c85-a5de-719730e28566-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7\" (UID: \"130c95ce-3eff-4c85-a5de-719730e28566\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:37.017188 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:37.017061 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/130c95ce-3eff-4c85-a5de-719730e28566-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7\" (UID: \"130c95ce-3eff-4c85-a5de-719730e28566\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:37.019295 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:37.019275 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/130c95ce-3eff-4c85-a5de-719730e28566-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7\" (UID: \"130c95ce-3eff-4c85-a5de-719730e28566\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:37.019685 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:37.019665 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/130c95ce-3eff-4c85-a5de-719730e28566-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7\" (UID: \"130c95ce-3eff-4c85-a5de-719730e28566\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:37.024470 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:37.024452 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8f8g\" (UniqueName: \"kubernetes.io/projected/130c95ce-3eff-4c85-a5de-719730e28566-kube-api-access-w8f8g\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7\" (UID: \"130c95ce-3eff-4c85-a5de-719730e28566\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:37.143527 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:37.143452 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:37.295522 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:37.295488 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7"] Jul 07 18:34:37.301219 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:34:37.301192 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod130c95ce_3eff_4c85_a5de_719730e28566.slice/crio-5289b6cda2b4d16342b11956db39a11dae698e8e08c827bb41edf5df122cce1e WatchSource:0}: Error finding container 5289b6cda2b4d16342b11956db39a11dae698e8e08c827bb41edf5df122cce1e: Status 404 returned error can't find the container with id 5289b6cda2b4d16342b11956db39a11dae698e8e08c827bb41edf5df122cce1e Jul 07 18:34:37.969408 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:37.969364 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" event={"ID":"130c95ce-3eff-4c85-a5de-719730e28566","Type":"ContainerStarted","Data":"0a5f7904e58d120ccdb247104d3a5b93a7612db0cd96c869f7af17ccb44f0632"} Jul 07 18:34:37.969839 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:37.969425 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" event={"ID":"130c95ce-3eff-4c85-a5de-719730e28566","Type":"ContainerStarted","Data":"5289b6cda2b4d16342b11956db39a11dae698e8e08c827bb41edf5df122cce1e"} Jul 07 18:34:37.988903 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:37.988859 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" podStartSLOduration=1.799012003 podStartE2EDuration="1.988844342s" totalImagesPullingTime="189.832339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:34:37.987639 +0000 UTC m=+701.371432396" watchObservedRunningTime="2026-07-07 18:34:37.988844342 +0000 UTC m=+701.372637781" Jul 07 18:34:39.238480 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:39.238448 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:34:47.143767 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:47.143681 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:47.143767 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:47.143733 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:47.156046 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:47.156018 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:48.023386 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:48.023360 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7" Jul 07 18:34:57.392148 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:34:57.392121 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:14.495684 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:14.495650 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:14.651550 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:14.651519 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:15.561275 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:15.561244 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-796cb9b767-vmhxm"] Jul 07 18:36:15.561695 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:15.561444 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-796cb9b767-vmhxm" podUID="724484d5-0bd8-44fa-abed-c67bbb1bd9a8" containerName="authorino" containerID="cri-o://a593a57324784ea9bfeba987c4f63a7ff4d9f0bcc563fd77b7ae18b7a4411071" gracePeriod=30 Jul 07 18:36:15.819301 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:15.819248 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-796cb9b767-vmhxm" Jul 07 18:36:15.940837 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:15.940798 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg5zg\" (UniqueName: \"kubernetes.io/projected/724484d5-0bd8-44fa-abed-c67bbb1bd9a8-kube-api-access-mg5zg\") pod \"724484d5-0bd8-44fa-abed-c67bbb1bd9a8\" (UID: \"724484d5-0bd8-44fa-abed-c67bbb1bd9a8\") " Jul 07 18:36:15.941031 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:15.940856 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/724484d5-0bd8-44fa-abed-c67bbb1bd9a8-tls-cert\") pod \"724484d5-0bd8-44fa-abed-c67bbb1bd9a8\" (UID: \"724484d5-0bd8-44fa-abed-c67bbb1bd9a8\") " Jul 07 18:36:15.942972 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:15.942947 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724484d5-0bd8-44fa-abed-c67bbb1bd9a8-kube-api-access-mg5zg" (OuterVolumeSpecName: "kube-api-access-mg5zg") pod "724484d5-0bd8-44fa-abed-c67bbb1bd9a8" (UID: "724484d5-0bd8-44fa-abed-c67bbb1bd9a8"). InnerVolumeSpecName "kube-api-access-mg5zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:36:15.951896 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:15.951872 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/724484d5-0bd8-44fa-abed-c67bbb1bd9a8-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "724484d5-0bd8-44fa-abed-c67bbb1bd9a8" (UID: "724484d5-0bd8-44fa-abed-c67bbb1bd9a8"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:36:15.977670 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:15.977637 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:16.041778 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:16.041745 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg5zg\" (UniqueName: \"kubernetes.io/projected/724484d5-0bd8-44fa-abed-c67bbb1bd9a8-kube-api-access-mg5zg\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:36:16.041778 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:16.041769 2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/724484d5-0bd8-44fa-abed-c67bbb1bd9a8-tls-cert\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:36:16.169042 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:16.168961 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:16.400771 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:16.400695 2578 generic.go:356] "Generic (PLEG): container finished" podID="724484d5-0bd8-44fa-abed-c67bbb1bd9a8" containerID="a593a57324784ea9bfeba987c4f63a7ff4d9f0bcc563fd77b7ae18b7a4411071" exitCode=0 Jul 07 18:36:16.400771 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:16.400736 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-796cb9b767-vmhxm" event={"ID":"724484d5-0bd8-44fa-abed-c67bbb1bd9a8","Type":"ContainerDied","Data":"a593a57324784ea9bfeba987c4f63a7ff4d9f0bcc563fd77b7ae18b7a4411071"} Jul 07 18:36:16.400997 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:16.400771 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-796cb9b767-vmhxm" Jul 07 18:36:16.400997 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:16.400783 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-796cb9b767-vmhxm" event={"ID":"724484d5-0bd8-44fa-abed-c67bbb1bd9a8","Type":"ContainerDied","Data":"c0e81533467ffec83d168236105f6555ac73071fec79c594312c033445e62ec1"} Jul 07 18:36:16.400997 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:16.400804 2578 scope.go:117] "RemoveContainer" containerID="a593a57324784ea9bfeba987c4f63a7ff4d9f0bcc563fd77b7ae18b7a4411071" Jul 07 18:36:16.409689 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:16.409671 2578 scope.go:117] "RemoveContainer" containerID="a593a57324784ea9bfeba987c4f63a7ff4d9f0bcc563fd77b7ae18b7a4411071" Jul 07 18:36:16.409981 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:36:16.409965 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a593a57324784ea9bfeba987c4f63a7ff4d9f0bcc563fd77b7ae18b7a4411071\": container with ID starting with a593a57324784ea9bfeba987c4f63a7ff4d9f0bcc563fd77b7ae18b7a4411071 not found: ID does not exist" containerID="a593a57324784ea9bfeba987c4f63a7ff4d9f0bcc563fd77b7ae18b7a4411071" Jul 07 18:36:16.410026 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:16.409990 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a593a57324784ea9bfeba987c4f63a7ff4d9f0bcc563fd77b7ae18b7a4411071"} err="failed to get container status \"a593a57324784ea9bfeba987c4f63a7ff4d9f0bcc563fd77b7ae18b7a4411071\": rpc error: code = NotFound desc = could not find container \"a593a57324784ea9bfeba987c4f63a7ff4d9f0bcc563fd77b7ae18b7a4411071\": container with ID starting with a593a57324784ea9bfeba987c4f63a7ff4d9f0bcc563fd77b7ae18b7a4411071 not found: ID does not exist" Jul 07 18:36:16.440492 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:16.440422 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-796cb9b767-vmhxm"] Jul 07 18:36:16.461269 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:16.461240 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-796cb9b767-vmhxm"] Jul 07 18:36:17.196869 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:17.196838 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="724484d5-0bd8-44fa-abed-c67bbb1bd9a8" path="/var/lib/kubelet/pods/724484d5-0bd8-44fa-abed-c67bbb1bd9a8/volumes" Jul 07 18:36:32.056082 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:32.056037 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:44.431083 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:44.431047 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:36:55.647494 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:36:55.647458 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:37:05.455507 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:37:05.455469 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:37:20.462558 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:37:20.462517 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:37:26.221704 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:37:26.221647 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-07T18:37:16Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-07T18:37:16Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-07T18:37:16Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-07T18:37:16Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-141-108.ec2.internal\": Patch \"https://aaeb12975e03d4bf48c0dfe118957bfe-6adc744d42e9c060.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-141-108.ec2.internal/status?timeout=10s\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 18:37:29.267020 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:37:29.266987 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:37:29.344893 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:37:29.344863 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:37:29.442202 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:37:29.442162 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:37:32.080364 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:37:32.080308 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:37:33.372233 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:37:33.372199 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:37:33.671584 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:37:33.671509 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:37:35.350796 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:37:35.350761 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:37:37.029028 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:37:37.028992 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:37:39.301853 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:37:39.301804 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:37:57.090161 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:37:57.090087 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 18:37:57.090662 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:37:57.090518 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 18:37:59.045136 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:37:59.045106 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:37:59.550485 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:37:59.550457 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:38:08.749557 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:38:08.749521 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:38:15.763923 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:38:15.763884 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:38:15.865620 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:38:15.865591 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:38:16.014611 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:38:16.014543 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:38:21.887228 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:38:21.887195 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:38:21.999130 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:38:21.999105 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:38:22.117499 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:38:22.117470 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:38:22.253885 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:38:22.253856 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:38:28.030147 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:38:28.030118 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:38:28.130526 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:38:28.130500 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:38:36.573041 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:38:36.573012 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:38:44.177483 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:38:44.177443 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:38:52.337550 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:38:52.337519 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:39:02.546065 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:39:02.546027 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:39:07.840244 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:39:07.840210 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:39:12.609254 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:39:12.609220 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:39:18.934593 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:39:18.934517 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:39:31.259564 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:39:31.259534 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:39:41.435279 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:39:41.435249 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:39:50.630566 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:39:50.630527 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:39:59.973006 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:39:59.972967 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:40:13.767737 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:40:13.767700 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:40:19.450756 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:40:19.450720 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:40:22.625793 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:40:22.625760 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:40:47.946590 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:40:47.946512 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:41:08.359455 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:41:08.359423 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:41:22.836455 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:41:22.836422 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:41:22.985252 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:41:22.985224 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:41:31.354415 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:41:31.354383 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:41:42.258129 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:41:42.258098 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:41:50.938980 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:41:50.938937 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:41:59.542633 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:41:59.542598 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:42:27.342999 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:27.342924 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26"] Jul 07 18:42:27.343508 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:27.343157 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26" podUID="c314533e-8a11-40dd-8549-77e5aa5e42a7" containerName="manager" containerID="cri-o://6cf8c1e905f583d71468dc0380b8c680db8d083b6c7bd4e31659870855138aa7" gracePeriod=10 Jul 07 18:42:29.171493 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:29.171462 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26" podUID="c314533e-8a11-40dd-8549-77e5aa5e42a7" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.31:8081/readyz\": dial tcp 10.134.0.31:8081: connect: connection refused" Jul 07 18:42:29.789655 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:29.789630 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26" Jul 07 18:42:29.858909 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:29.858881 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctkjf\" (UniqueName: \"kubernetes.io/projected/c314533e-8a11-40dd-8549-77e5aa5e42a7-kube-api-access-ctkjf\") pod \"c314533e-8a11-40dd-8549-77e5aa5e42a7\" (UID: \"c314533e-8a11-40dd-8549-77e5aa5e42a7\") " Jul 07 18:42:29.859030 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:29.858919 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c314533e-8a11-40dd-8549-77e5aa5e42a7-extensions-socket-volume\") pod \"c314533e-8a11-40dd-8549-77e5aa5e42a7\" (UID: \"c314533e-8a11-40dd-8549-77e5aa5e42a7\") " Jul 07 18:42:29.859395 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:29.859369 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c314533e-8a11-40dd-8549-77e5aa5e42a7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c314533e-8a11-40dd-8549-77e5aa5e42a7" (UID: "c314533e-8a11-40dd-8549-77e5aa5e42a7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:42:29.860765 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:29.860739 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c314533e-8a11-40dd-8549-77e5aa5e42a7-kube-api-access-ctkjf" (OuterVolumeSpecName: "kube-api-access-ctkjf") pod "c314533e-8a11-40dd-8549-77e5aa5e42a7" (UID: "c314533e-8a11-40dd-8549-77e5aa5e42a7"). InnerVolumeSpecName "kube-api-access-ctkjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:42:29.959719 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:29.959663 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctkjf\" (UniqueName: \"kubernetes.io/projected/c314533e-8a11-40dd-8549-77e5aa5e42a7-kube-api-access-ctkjf\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:42:29.959719 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:29.959686 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c314533e-8a11-40dd-8549-77e5aa5e42a7-extensions-socket-volume\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:42:30.058328 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:30.058295 2578 generic.go:356] "Generic (PLEG): container finished" podID="c314533e-8a11-40dd-8549-77e5aa5e42a7" containerID="6cf8c1e905f583d71468dc0380b8c680db8d083b6c7bd4e31659870855138aa7" exitCode=0 Jul 07 18:42:30.058419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:30.058347 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26" event={"ID":"c314533e-8a11-40dd-8549-77e5aa5e42a7","Type":"ContainerDied","Data":"6cf8c1e905f583d71468dc0380b8c680db8d083b6c7bd4e31659870855138aa7"} Jul 07 18:42:30.058419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:30.058356 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26" Jul 07 18:42:30.058419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:30.058389 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26" event={"ID":"c314533e-8a11-40dd-8549-77e5aa5e42a7","Type":"ContainerDied","Data":"fbd0a87f097dd3241f0affe0225b631bc22417d3d519b525202bcf395c735582"} Jul 07 18:42:30.058419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:30.058410 2578 scope.go:117] "RemoveContainer" containerID="6cf8c1e905f583d71468dc0380b8c680db8d083b6c7bd4e31659870855138aa7" Jul 07 18:42:30.067526 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:30.067509 2578 scope.go:117] "RemoveContainer" containerID="6cf8c1e905f583d71468dc0380b8c680db8d083b6c7bd4e31659870855138aa7" Jul 07 18:42:30.067801 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:42:30.067777 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf8c1e905f583d71468dc0380b8c680db8d083b6c7bd4e31659870855138aa7\": container with ID starting with 6cf8c1e905f583d71468dc0380b8c680db8d083b6c7bd4e31659870855138aa7 not found: ID does not exist" containerID="6cf8c1e905f583d71468dc0380b8c680db8d083b6c7bd4e31659870855138aa7" Jul 07 18:42:30.067857 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:30.067807 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf8c1e905f583d71468dc0380b8c680db8d083b6c7bd4e31659870855138aa7"} err="failed to get container status \"6cf8c1e905f583d71468dc0380b8c680db8d083b6c7bd4e31659870855138aa7\": rpc error: code = NotFound desc = could not find container \"6cf8c1e905f583d71468dc0380b8c680db8d083b6c7bd4e31659870855138aa7\": container with ID starting with 6cf8c1e905f583d71468dc0380b8c680db8d083b6c7bd4e31659870855138aa7 not found: ID does not exist" Jul 07 18:42:30.084671 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:30.084645 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26"] Jul 07 18:42:30.086640 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:30.086623 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kkw26"] Jul 07 18:42:31.196724 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:31.196695 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c314533e-8a11-40dd-8549-77e5aa5e42a7" path="/var/lib/kubelet/pods/c314533e-8a11-40dd-8549-77e5aa5e42a7/volumes" Jul 07 18:42:57.116833 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:57.116807 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 18:42:57.118565 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:42:57.118543 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 18:43:28.541462 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:28.541427 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lsfhk"] Jul 07 18:43:28.542068 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:28.541890 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="724484d5-0bd8-44fa-abed-c67bbb1bd9a8" containerName="authorino" Jul 07 18:43:28.542068 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:28.541908 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="724484d5-0bd8-44fa-abed-c67bbb1bd9a8" containerName="authorino" Jul 07 18:43:28.542068 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:28.541945 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c314533e-8a11-40dd-8549-77e5aa5e42a7" containerName="manager" Jul 07 18:43:28.542068 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:28.541954 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c314533e-8a11-40dd-8549-77e5aa5e42a7" containerName="manager" Jul 07 18:43:28.542068 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:28.542045 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="724484d5-0bd8-44fa-abed-c67bbb1bd9a8" containerName="authorino" Jul 07 18:43:28.542068 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:28.542056 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c314533e-8a11-40dd-8549-77e5aa5e42a7" containerName="manager" Jul 07 18:43:28.544985 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:28.544964 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lsfhk" Jul 07 18:43:28.548152 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:28.548131 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p2v2g\"" Jul 07 18:43:28.568659 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:28.568632 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lsfhk"] Jul 07 18:43:28.607920 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:28.607886 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwsfz\" (UniqueName: \"kubernetes.io/projected/071b8ef2-d40c-40e5-8f28-3bfa998f1bd1-kube-api-access-jwsfz\") pod \"kuadrant-operator-controller-manager-86878c769f-lsfhk\" (UID: \"071b8ef2-d40c-40e5-8f28-3bfa998f1bd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lsfhk" Jul 07 18:43:28.607920 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:28.607923 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/071b8ef2-d40c-40e5-8f28-3bfa998f1bd1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lsfhk\" (UID: \"071b8ef2-d40c-40e5-8f28-3bfa998f1bd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lsfhk" Jul 07 18:43:28.709192 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:28.709155 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwsfz\" (UniqueName: \"kubernetes.io/projected/071b8ef2-d40c-40e5-8f28-3bfa998f1bd1-kube-api-access-jwsfz\") pod \"kuadrant-operator-controller-manager-86878c769f-lsfhk\" (UID: \"071b8ef2-d40c-40e5-8f28-3bfa998f1bd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lsfhk" Jul 07 18:43:28.709192 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:28.709197 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/071b8ef2-d40c-40e5-8f28-3bfa998f1bd1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lsfhk\" (UID: \"071b8ef2-d40c-40e5-8f28-3bfa998f1bd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lsfhk" Jul 07 18:43:28.709585 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:28.709569 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/071b8ef2-d40c-40e5-8f28-3bfa998f1bd1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lsfhk\" (UID: \"071b8ef2-d40c-40e5-8f28-3bfa998f1bd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lsfhk" Jul 07 18:43:28.730166 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:28.730132 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwsfz\" (UniqueName: \"kubernetes.io/projected/071b8ef2-d40c-40e5-8f28-3bfa998f1bd1-kube-api-access-jwsfz\") pod \"kuadrant-operator-controller-manager-86878c769f-lsfhk\" (UID: \"071b8ef2-d40c-40e5-8f28-3bfa998f1bd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lsfhk" Jul 07 18:43:28.854683 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:28.854597 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lsfhk" Jul 07 18:43:29.014006 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:29.013982 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lsfhk"] Jul 07 18:43:29.016374 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:43:29.016351 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod071b8ef2_d40c_40e5_8f28_3bfa998f1bd1.slice/crio-a312a2961bbfd41de2d8571b4a8c84a5cb1aaf08ff5d79d9c83d82a98d0a55e7 WatchSource:0}: Error finding container a312a2961bbfd41de2d8571b4a8c84a5cb1aaf08ff5d79d9c83d82a98d0a55e7: Status 404 returned error can't find the container with id a312a2961bbfd41de2d8571b4a8c84a5cb1aaf08ff5d79d9c83d82a98d0a55e7 Jul 07 18:43:29.311387 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:29.311351 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lsfhk" event={"ID":"071b8ef2-d40c-40e5-8f28-3bfa998f1bd1","Type":"ContainerStarted","Data":"1cb24394c626bee589707d5708c6aa2a5e68915053b3703de14aae736bcf3ebf"} Jul 07 18:43:29.311586 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:29.311406 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lsfhk" event={"ID":"071b8ef2-d40c-40e5-8f28-3bfa998f1bd1","Type":"ContainerStarted","Data":"a312a2961bbfd41de2d8571b4a8c84a5cb1aaf08ff5d79d9c83d82a98d0a55e7"} Jul 07 18:43:29.311586 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:29.311456 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lsfhk" Jul 07 18:43:29.339606 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:29.339563 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lsfhk" podStartSLOduration=1.339546535 podStartE2EDuration="1.339546535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:43:29.338661394 +0000 UTC m=+1232.722454821" watchObservedRunningTime="2026-07-07 18:43:29.339546535 +0000 UTC m=+1232.723339912" Jul 07 18:43:40.318967 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:40.318936 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lsfhk" Jul 07 18:43:49.137939 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:43:49.137828 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:44:00.173236 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:44:00.173199 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:44:08.242230 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:44:08.242201 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:44:16.832399 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:44:16.832367 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:44:26.447602 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:44:26.447562 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:44:34.059990 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:44:34.059955 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:44:44.129136 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:44:44.129103 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:44:55.835337 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:44:55.835283 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:45:03.038353 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:45:03.038310 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:45:11.249798 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:45:11.249765 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:45:18.536571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:45:18.536496 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:45:20.284525 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:45:20.284492 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:45:21.438546 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:45:21.438511 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:45:26.350873 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:45:26.350833 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:45:32.235205 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:45:32.235170 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:45:55.331635 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:45:55.331596 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:46:04.535306 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:46:04.535277 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:46:15.429152 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:46:15.429117 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:46:23.842568 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:46:23.842530 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:46:29.835230 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:46:29.835192 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:46:45.443403 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:46:45.443370 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:46:56.352079 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:46:56.351994 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:46:58.875622 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:46:58.875589 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:47:04.832556 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:47:04.832525 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:47:15.347794 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:47:15.347768 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:47:24.730439 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:47:24.730401 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:47:33.144027 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:47:33.143994 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:47:38.257216 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:47:38.257188 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:47:47.766232 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:47:47.766197 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:47:57.043195 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:47:57.043153 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:47:57.140463 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:47:57.140436 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 18:47:57.141998 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:47:57.141976 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 18:48:07.058026 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:48:07.057991 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:48:18.774614 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:48:18.774534 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:48:22.767677 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:48:22.767640 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:48:30.848422 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:48:30.848388 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:48:39.649856 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:48:39.649826 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:48:50.859532 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:48:50.856542 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:49:03.348398 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:03.348363 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:49:15.638071 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:15.638042 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:49:18.844394 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:18.844361 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:24.147978 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:24.147943 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:49:28.644928 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:28.644899 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:36.932993 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:36.932957 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:37.769675 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:37.769643 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:49:42.325767 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:42.325740 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:45.482001 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:45.481969 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:45.632394 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:45.632357 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:45.698374 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:45.698342 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:49:46.035551 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:46.035520 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:52.424830 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.424798 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:52.524733 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.524696 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:52.666986 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.666961 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv"] Jul 07 18:49:52.670052 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.670018 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.672627 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.672605 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4110137c-openshift-default-dockercfg-p5pdk\"" Jul 07 18:49:52.700519 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.700491 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv"] Jul 07 18:49:52.712076 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.712056 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:52.807779 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.807755 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:52.838523 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.838502 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-token\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.838609 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.838528 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-data\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.838609 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.838546 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-workload-certs\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.838682 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.838643 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-workload-socket\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.838682 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.838676 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istiod-ca-cert\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.838744 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.838703 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-podinfo\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.838744 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.838726 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-credential-socket\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.838810 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.838790 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-envoy\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.838843 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.838827 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n49pq\" (UniqueName: \"kubernetes.io/projected/978fd570-636d-401e-a3c8-4fbf4a02fb3b-kube-api-access-n49pq\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.939360 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.939278 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-workload-socket\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.939360 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.939314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istiod-ca-cert\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.939360 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.939352 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-podinfo\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.939557 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.939379 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-credential-socket\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.939557 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.939410 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-envoy\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.939557 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.939434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n49pq\" (UniqueName: \"kubernetes.io/projected/978fd570-636d-401e-a3c8-4fbf4a02fb3b-kube-api-access-n49pq\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.939557 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.939467 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-token\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.939557 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.939495 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-data\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.939557 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.939522 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-workload-certs\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.939872 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.939682 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-workload-socket\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.939872 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.939734 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-credential-socket\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.940073 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.939887 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-data\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.940073 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.939892 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-workload-certs\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.940232 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.940215 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istiod-ca-cert\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.941699 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.941679 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-envoy\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.941953 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.941937 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-podinfo\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.957839 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.957806 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-token\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.957941 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.957872 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n49pq\" (UniqueName: \"kubernetes.io/projected/978fd570-636d-401e-a3c8-4fbf4a02fb3b-kube-api-access-n49pq\") pod \"e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:52.980223 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:52.980201 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:53.144591 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:53.144551 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv"] Jul 07 18:49:53.156336 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:49:53.156285 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978fd570_636d_401e_a3c8_4fbf4a02fb3b.slice/crio-45c3d302a138a12331cb20a38d4bb584aa360954b4bbdd534274922270c3092e WatchSource:0}: Error finding container 45c3d302a138a12331cb20a38d4bb584aa360954b4bbdd534274922270c3092e: Status 404 returned error can't find the container with id 45c3d302a138a12331cb20a38d4bb584aa360954b4bbdd534274922270c3092e Jul 07 18:49:53.157971 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:53.157928 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:49:53.158094 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:53.158042 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:49:53.158094 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:53.158088 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:49:53.920303 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:53.920260 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:53.972599 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:53.972563 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" event={"ID":"978fd570-636d-401e-a3c8-4fbf4a02fb3b","Type":"ContainerStarted","Data":"919e9a4ce4877797b46cfed85655df594961f3a620d958dcec538503083efc1e"} Jul 07 18:49:53.972770 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:53.972624 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" event={"ID":"978fd570-636d-401e-a3c8-4fbf4a02fb3b","Type":"ContainerStarted","Data":"45c3d302a138a12331cb20a38d4bb584aa360954b4bbdd534274922270c3092e"} Jul 07 18:49:53.980712 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:53.980685 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:54.012298 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:54.012250 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" podStartSLOduration=2.012208944 podStartE2EDuration="2.012208944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:49:54.01013957 +0000 UTC m=+1617.393932976" watchObservedRunningTime="2026-07-07 18:49:54.012208944 +0000 UTC m=+1617.396002339" Jul 07 18:49:54.044287 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:54.044253 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:54.192083 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:54.192003 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:54.337945 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:54.337897 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:54.850500 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:54.850466 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:54.985278 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:54.985245 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:54.986664 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:54.986634 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:55.981625 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:55.981595 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:55.982474 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:55.982456 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:49:56.677860 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:56.677829 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:58.452337 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:58.452301 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:49:59.060288 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:59.060260 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:59.247007 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:59.246973 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv"] Jul 07 18:49:59.247178 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:59.247152 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" podUID="978fd570-636d-401e-a3c8-4fbf4a02fb3b" containerName="istio-proxy" containerID="cri-o://919e9a4ce4877797b46cfed85655df594961f3a620d958dcec538503083efc1e" gracePeriod=30 Jul 07 18:49:59.280791 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:59.280753 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:59.366984 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:59.366899 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:49:59.617588 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:49:59.617503 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:01.601855 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:01.601800 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:05.736510 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:05.736472 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:05.866681 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:05.866656 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:06.025732 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.025704 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:06.033436 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.033409 2578 generic.go:356] "Generic (PLEG): container finished" podID="978fd570-636d-401e-a3c8-4fbf4a02fb3b" containerID="919e9a4ce4877797b46cfed85655df594961f3a620d958dcec538503083efc1e" exitCode=0 Jul 07 18:50:06.033562 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.033471 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" event={"ID":"978fd570-636d-401e-a3c8-4fbf4a02fb3b","Type":"ContainerDied","Data":"919e9a4ce4877797b46cfed85655df594961f3a620d958dcec538503083efc1e"} Jul 07 18:50:06.102195 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.102145 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:50:06.140397 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.140367 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:06.154503 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.154481 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istiod-ca-cert\") pod \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " Jul 07 18:50:06.154610 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.154520 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-data\") pod \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " Jul 07 18:50:06.154610 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.154542 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-token\") pod \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " Jul 07 18:50:06.154610 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.154575 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-workload-socket\") pod \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " Jul 07 18:50:06.154610 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.154598 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-podinfo\") pod \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " Jul 07 18:50:06.154845 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.154633 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-workload-certs\") pod \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " Jul 07 18:50:06.154845 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.154665 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-credential-socket\") pod \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " Jul 07 18:50:06.154845 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.154757 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-envoy\") pod \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " Jul 07 18:50:06.154845 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.154811 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n49pq\" (UniqueName: \"kubernetes.io/projected/978fd570-636d-401e-a3c8-4fbf4a02fb3b-kube-api-access-n49pq\") pod \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\" (UID: \"978fd570-636d-401e-a3c8-4fbf4a02fb3b\") " Jul 07 18:50:06.155039 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.154901 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "978fd570-636d-401e-a3c8-4fbf4a02fb3b" (UID: "978fd570-636d-401e-a3c8-4fbf4a02fb3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:50:06.155097 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.155010 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "978fd570-636d-401e-a3c8-4fbf4a02fb3b" (UID: "978fd570-636d-401e-a3c8-4fbf4a02fb3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:50:06.155097 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.155076 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "978fd570-636d-401e-a3c8-4fbf4a02fb3b" (UID: "978fd570-636d-401e-a3c8-4fbf4a02fb3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:50:06.155213 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.155129 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-credential-socket\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:50:06.155213 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.155147 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istiod-ca-cert\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:50:06.155346 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.155291 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "978fd570-636d-401e-a3c8-4fbf4a02fb3b" (UID: "978fd570-636d-401e-a3c8-4fbf4a02fb3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:50:06.155442 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.155423 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-data" (OuterVolumeSpecName: "istio-data") pod "978fd570-636d-401e-a3c8-4fbf4a02fb3b" (UID: "978fd570-636d-401e-a3c8-4fbf4a02fb3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:50:06.156928 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.156905 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "978fd570-636d-401e-a3c8-4fbf4a02fb3b" (UID: "978fd570-636d-401e-a3c8-4fbf4a02fb3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:50:06.157037 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.157008 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "978fd570-636d-401e-a3c8-4fbf4a02fb3b" (UID: "978fd570-636d-401e-a3c8-4fbf4a02fb3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:50:06.157214 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.157198 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978fd570-636d-401e-a3c8-4fbf4a02fb3b-kube-api-access-n49pq" (OuterVolumeSpecName: "kube-api-access-n49pq") pod "978fd570-636d-401e-a3c8-4fbf4a02fb3b" (UID: "978fd570-636d-401e-a3c8-4fbf4a02fb3b"). InnerVolumeSpecName "kube-api-access-n49pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:50:06.157265 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.157208 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-token" (OuterVolumeSpecName: "istio-token") pod "978fd570-636d-401e-a3c8-4fbf4a02fb3b" (UID: "978fd570-636d-401e-a3c8-4fbf4a02fb3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:50:06.256429 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.256401 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n49pq\" (UniqueName: \"kubernetes.io/projected/978fd570-636d-401e-a3c8-4fbf4a02fb3b-kube-api-access-n49pq\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:50:06.256429 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.256427 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-data\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:50:06.256573 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.256441 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-token\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:50:06.256573 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.256453 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-workload-socket\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:50:06.256573 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.256464 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-podinfo\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:50:06.256573 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.256477 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-workload-certs\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:50:06.256573 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:06.256489 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978fd570-636d-401e-a3c8-4fbf4a02fb3b-istio-envoy\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:50:07.039436 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:07.039400 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" event={"ID":"978fd570-636d-401e-a3c8-4fbf4a02fb3b","Type":"ContainerDied","Data":"45c3d302a138a12331cb20a38d4bb584aa360954b4bbdd534274922270c3092e"} Jul 07 18:50:07.039896 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:07.039451 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv" Jul 07 18:50:07.039896 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:07.039460 2578 scope.go:117] "RemoveContainer" containerID="919e9a4ce4877797b46cfed85655df594961f3a620d958dcec538503083efc1e" Jul 07 18:50:07.063514 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:07.063491 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv"] Jul 07 18:50:07.072057 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:07.072036 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4110137c-openshift-default-776f878fbc-xkzhv"] Jul 07 18:50:07.197417 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:07.197391 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="978fd570-636d-401e-a3c8-4fbf4a02fb3b" path="/var/lib/kubelet/pods/978fd570-636d-401e-a3c8-4fbf4a02fb3b/volumes" Jul 07 18:50:07.205800 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:07.205776 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:07.349871 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:07.349796 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:07.460522 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:07.460489 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:07.868633 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:07.868601 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:08.195185 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:08.195099 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:08.323261 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:08.323230 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:08.547982 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:08.547948 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:50:09.705152 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:09.705120 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:12.354838 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:12.354805 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:12.876477 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:12.876428 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:18.673953 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:18.673915 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:18.788781 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:18.788748 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:21.040288 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:21.040258 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:25.075141 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.075108 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:25.322613 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.322586 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg"] Jul 07 18:50:25.322924 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.322912 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="978fd570-636d-401e-a3c8-4fbf4a02fb3b" containerName="istio-proxy" Jul 07 18:50:25.322966 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.322926 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="978fd570-636d-401e-a3c8-4fbf4a02fb3b" containerName="istio-proxy" Jul 07 18:50:25.323005 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.322996 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="978fd570-636d-401e-a3c8-4fbf4a02fb3b" containerName="istio-proxy" Jul 07 18:50:25.327398 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.327350 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.330526 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.330507 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-593c94a9-openshift-default-dockercfg-mw2z4\"" Jul 07 18:50:25.346681 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.346659 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg"] Jul 07 18:50:25.414830 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.414791 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-credential-socket\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.415017 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.414838 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-data\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.415017 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.414876 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-envoy\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.415017 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.414933 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c0545b-2456-4a1e-8f49-531b5063c69f-istiod-ca-cert\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.415017 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.414979 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q2wp\" (UniqueName: \"kubernetes.io/projected/b4c0545b-2456-4a1e-8f49-531b5063c69f-kube-api-access-9q2wp\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.415017 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.415009 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-workload-certs\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.415240 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.415037 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-podinfo\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.415240 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.415068 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-token\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.415240 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.415141 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-workload-socket\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.516070 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.516018 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-envoy\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.516369 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.516349 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c0545b-2456-4a1e-8f49-531b5063c69f-istiod-ca-cert\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.516508 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.516494 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q2wp\" (UniqueName: \"kubernetes.io/projected/b4c0545b-2456-4a1e-8f49-531b5063c69f-kube-api-access-9q2wp\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.516637 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.516624 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-workload-certs\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.516744 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.516731 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-podinfo\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.516854 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.516842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-token\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.516980 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.516965 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-workload-socket\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.517090 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.517076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-credential-socket\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.517197 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.517184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-data\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.517736 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.517713 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-data\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.521031 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.520535 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-envoy\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.522070 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.521389 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c0545b-2456-4a1e-8f49-531b5063c69f-istiod-ca-cert\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.522194 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.522069 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-workload-certs\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.522516 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.522394 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-workload-socket\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.522858 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.522835 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-credential-socket\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.524678 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.524655 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-podinfo\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.530253 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.530233 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q2wp\" (UniqueName: \"kubernetes.io/projected/b4c0545b-2456-4a1e-8f49-531b5063c69f-kube-api-access-9q2wp\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.530442 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.530425 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-token\") pod \"e2e-derive-593c94a9-openshift-default-767d879479-4wctg\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.637290 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.637198 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:25.798387 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.798358 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg"] Jul 07 18:50:25.800833 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:50:25.800805 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c0545b_2456_4a1e_8f49_531b5063c69f.slice/crio-9ff355db5bcf42684d845995bd82f3a281723ac8b302bbf87acf3e7a87c4d322 WatchSource:0}: Error finding container 9ff355db5bcf42684d845995bd82f3a281723ac8b302bbf87acf3e7a87c4d322: Status 404 returned error can't find the container with id 9ff355db5bcf42684d845995bd82f3a281723ac8b302bbf87acf3e7a87c4d322 Jul 07 18:50:25.802784 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.802758 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:50:25.802845 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.802817 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:50:25.802923 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:25.802846 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:50:26.130808 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:26.130775 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" event={"ID":"b4c0545b-2456-4a1e-8f49-531b5063c69f","Type":"ContainerStarted","Data":"70239694c8352aeb51476413c6952e45f373c4484ea3f42ea51d37403489469d"} Jul 07 18:50:26.131315 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:26.130835 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" event={"ID":"b4c0545b-2456-4a1e-8f49-531b5063c69f","Type":"ContainerStarted","Data":"9ff355db5bcf42684d845995bd82f3a281723ac8b302bbf87acf3e7a87c4d322"} Jul 07 18:50:26.171297 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:26.171248 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" podStartSLOduration=1.171232035 podStartE2EDuration="1.171232035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:50:26.166125214 +0000 UTC m=+1649.549918610" watchObservedRunningTime="2026-07-07 18:50:26.171232035 +0000 UTC m=+1649.555025430" Jul 07 18:50:26.365533 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:26.365504 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:26.637708 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:26.637668 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:26.642854 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:26.642830 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:26.651423 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:26.651393 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:26.809912 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:26.809885 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:26.919894 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:26.919810 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:27.111826 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:27.111794 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:27.135843 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:27.135822 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:27.136826 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:27.136807 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:27.598630 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:27.598602 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:27.698533 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:27.698504 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:29.015119 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:29.015088 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:29.449520 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:29.449489 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:50:30.972193 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:30.972148 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:31.226733 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:31.226642 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg"] Jul 07 18:50:31.226904 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:31.226866 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" podUID="b4c0545b-2456-4a1e-8f49-531b5063c69f" containerName="istio-proxy" containerID="cri-o://70239694c8352aeb51476413c6952e45f373c4484ea3f42ea51d37403489469d" gracePeriod=30 Jul 07 18:50:31.376993 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:31.376913 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:31.521605 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:31.521507 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:31.793506 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:31.793388 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:32.800586 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:32.800541 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:37.183294 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.183259 2578 generic.go:356] "Generic (PLEG): container finished" podID="b4c0545b-2456-4a1e-8f49-531b5063c69f" containerID="70239694c8352aeb51476413c6952e45f373c4484ea3f42ea51d37403489469d" exitCode=0 Jul 07 18:50:37.183694 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.183305 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" event={"ID":"b4c0545b-2456-4a1e-8f49-531b5063c69f","Type":"ContainerDied","Data":"70239694c8352aeb51476413c6952e45f373c4484ea3f42ea51d37403489469d"} Jul 07 18:50:37.282796 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.282767 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:37.440106 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.440015 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-credential-socket\") pod \"b4c0545b-2456-4a1e-8f49-531b5063c69f\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " Jul 07 18:50:37.440106 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.440093 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-data\") pod \"b4c0545b-2456-4a1e-8f49-531b5063c69f\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " Jul 07 18:50:37.440406 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.440125 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-workload-socket\") pod \"b4c0545b-2456-4a1e-8f49-531b5063c69f\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " Jul 07 18:50:37.440406 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.440147 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-token\") pod \"b4c0545b-2456-4a1e-8f49-531b5063c69f\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " Jul 07 18:50:37.440406 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.440168 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-workload-certs\") pod \"b4c0545b-2456-4a1e-8f49-531b5063c69f\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " Jul 07 18:50:37.440406 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.440193 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q2wp\" (UniqueName: \"kubernetes.io/projected/b4c0545b-2456-4a1e-8f49-531b5063c69f-kube-api-access-9q2wp\") pod \"b4c0545b-2456-4a1e-8f49-531b5063c69f\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " Jul 07 18:50:37.440406 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.440240 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-envoy\") pod \"b4c0545b-2456-4a1e-8f49-531b5063c69f\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " Jul 07 18:50:37.440406 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.440259 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c0545b-2456-4a1e-8f49-531b5063c69f-istiod-ca-cert\") pod \"b4c0545b-2456-4a1e-8f49-531b5063c69f\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " Jul 07 18:50:37.440406 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.440347 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-podinfo\") pod \"b4c0545b-2456-4a1e-8f49-531b5063c69f\" (UID: \"b4c0545b-2456-4a1e-8f49-531b5063c69f\") " Jul 07 18:50:37.440406 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.440390 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b4c0545b-2456-4a1e-8f49-531b5063c69f" (UID: "b4c0545b-2456-4a1e-8f49-531b5063c69f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:50:37.440864 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.440591 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b4c0545b-2456-4a1e-8f49-531b5063c69f" (UID: "b4c0545b-2456-4a1e-8f49-531b5063c69f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:50:37.440864 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.440602 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-credential-socket\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:50:37.440864 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.440690 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c0545b-2456-4a1e-8f49-531b5063c69f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b4c0545b-2456-4a1e-8f49-531b5063c69f" (UID: "b4c0545b-2456-4a1e-8f49-531b5063c69f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:50:37.440864 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.440791 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b4c0545b-2456-4a1e-8f49-531b5063c69f" (UID: "b4c0545b-2456-4a1e-8f49-531b5063c69f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:50:37.441191 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.441166 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-data" (OuterVolumeSpecName: "istio-data") pod "b4c0545b-2456-4a1e-8f49-531b5063c69f" (UID: "b4c0545b-2456-4a1e-8f49-531b5063c69f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:50:37.442645 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.442614 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b4c0545b-2456-4a1e-8f49-531b5063c69f" (UID: "b4c0545b-2456-4a1e-8f49-531b5063c69f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:50:37.442740 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.442677 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b4c0545b-2456-4a1e-8f49-531b5063c69f" (UID: "b4c0545b-2456-4a1e-8f49-531b5063c69f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:50:37.442921 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.442901 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-token" (OuterVolumeSpecName: "istio-token") pod "b4c0545b-2456-4a1e-8f49-531b5063c69f" (UID: "b4c0545b-2456-4a1e-8f49-531b5063c69f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:50:37.442998 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.442896 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c0545b-2456-4a1e-8f49-531b5063c69f-kube-api-access-9q2wp" (OuterVolumeSpecName: "kube-api-access-9q2wp") pod "b4c0545b-2456-4a1e-8f49-531b5063c69f" (UID: "b4c0545b-2456-4a1e-8f49-531b5063c69f"). InnerVolumeSpecName "kube-api-access-9q2wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:50:37.541840 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.541814 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-workload-certs\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:50:37.541840 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.541836 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q2wp\" (UniqueName: \"kubernetes.io/projected/b4c0545b-2456-4a1e-8f49-531b5063c69f-kube-api-access-9q2wp\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:50:37.541840 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.541847 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-envoy\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:50:37.542061 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.541856 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c0545b-2456-4a1e-8f49-531b5063c69f-istiod-ca-cert\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:50:37.542061 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.541863 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-podinfo\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:50:37.542061 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.541870 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-data\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:50:37.542061 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.541878 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c0545b-2456-4a1e-8f49-531b5063c69f-workload-socket\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:50:37.542061 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.541885 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c0545b-2456-4a1e-8f49-531b5063c69f-istio-token\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:50:37.561246 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.561221 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:37.658493 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:37.658464 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:38.098891 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:38.098857 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:38.188776 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:38.188748 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" Jul 07 18:50:38.189167 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:38.188752 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg" event={"ID":"b4c0545b-2456-4a1e-8f49-531b5063c69f","Type":"ContainerDied","Data":"9ff355db5bcf42684d845995bd82f3a281723ac8b302bbf87acf3e7a87c4d322"} Jul 07 18:50:38.189167 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:38.188869 2578 scope.go:117] "RemoveContainer" containerID="70239694c8352aeb51476413c6952e45f373c4484ea3f42ea51d37403489469d" Jul 07 18:50:38.202201 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:38.202168 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:38.215213 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:38.215192 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg"] Jul 07 18:50:38.218960 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:38.218941 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-593c94a9-openshift-default-767d879479-4wctg"] Jul 07 18:50:38.392148 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:38.392062 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:39.197726 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:39.197684 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4c0545b-2456-4a1e-8f49-531b5063c69f" path="/var/lib/kubelet/pods/b4c0545b-2456-4a1e-8f49-531b5063c69f/volumes" Jul 07 18:50:40.027929 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:40.027899 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:41.546381 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:41.546350 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:50:46.650846 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:46.650811 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:46.775734 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:46.775705 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:46.886366 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:46.886295 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:46.976365 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:46.976337 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:50:47.489242 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:50:47.489212 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:51:13.554435 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:51:13.554406 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:51:20.008270 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:51:20.008190 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:51:20.194959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:51:20.194927 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:51:20.333706 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:51:20.333627 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:51:29.858657 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:51:29.858624 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:51:36.209301 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:51:36.209241 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:51:42.902912 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:51:42.902873 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:51:43.012702 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:51:43.012671 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:51:43.108760 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:51:43.108735 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:51:53.694681 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:51:53.694644 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:51:54.573521 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:51:54.573491 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:00.668512 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:00.668480 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:00.789012 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:00.788976 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:00.940052 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:00.939975 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:01.046003 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:01.045970 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:02.053504 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:02.053465 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:02.316734 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:02.316655 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:02.973561 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:02.973530 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:52:04.062947 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:04.062917 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:10.354260 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:10.354230 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:52:10.977518 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:10.977482 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:11.064477 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:11.064444 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:11.729569 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:11.729516 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:17.953715 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:17.953683 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:52:18.101038 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.101009 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:18.201558 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.201528 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:18.293137 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.293110 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:18.427155 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.427123 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:18.779333 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.779296 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx"] Jul 07 18:52:18.779640 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.779627 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c0545b-2456-4a1e-8f49-531b5063c69f" containerName="istio-proxy" Jul 07 18:52:18.779708 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.779642 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c0545b-2456-4a1e-8f49-531b5063c69f" containerName="istio-proxy" Jul 07 18:52:18.779708 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.779688 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4c0545b-2456-4a1e-8f49-531b5063c69f" containerName="istio-proxy" Jul 07 18:52:18.782706 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.782689 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.786334 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.786302 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8c3df14f-openshift-default-dockercfg-l8qmj\"" Jul 07 18:52:18.799996 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.799976 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-workload-certs\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.800085 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.800005 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-istio-data\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.800085 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.800029 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ade3fec-8e2a-4840-9390-43d3316df678-istio-token\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.800085 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.800049 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ade3fec-8e2a-4840-9390-43d3316df678-istio-podinfo\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.800228 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.800101 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nkv9\" (UniqueName: \"kubernetes.io/projected/3ade3fec-8e2a-4840-9390-43d3316df678-kube-api-access-9nkv9\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.800228 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.800167 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-istio-envoy\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.800228 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.800207 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-workload-socket\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.800398 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.800239 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-credential-socket\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.800398 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.800278 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ade3fec-8e2a-4840-9390-43d3316df678-istiod-ca-cert\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.830052 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.830012 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx"] Jul 07 18:52:18.843509 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.843480 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:18.901592 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.901572 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ade3fec-8e2a-4840-9390-43d3316df678-istio-podinfo\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.901701 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.901598 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nkv9\" (UniqueName: \"kubernetes.io/projected/3ade3fec-8e2a-4840-9390-43d3316df678-kube-api-access-9nkv9\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.901701 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.901624 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-istio-envoy\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.901852 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.901833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-workload-socket\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.901978 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.901949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-credential-socket\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.902057 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.902003 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ade3fec-8e2a-4840-9390-43d3316df678-istiod-ca-cert\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.902112 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.902074 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-workload-certs\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.902172 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.902110 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-istio-data\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.902172 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.902152 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ade3fec-8e2a-4840-9390-43d3316df678-istio-token\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.902282 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.902189 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-workload-socket\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.902282 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.902244 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-credential-socket\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.902475 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.902454 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-istio-data\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.902750 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.902728 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-workload-certs\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.902866 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.902823 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ade3fec-8e2a-4840-9390-43d3316df678-istiod-ca-cert\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.904041 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.904018 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-istio-envoy\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.904530 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.904510 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ade3fec-8e2a-4840-9390-43d3316df678-istio-podinfo\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.919986 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.919963 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ade3fec-8e2a-4840-9390-43d3316df678-istio-token\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.924435 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.924412 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nkv9\" (UniqueName: \"kubernetes.io/projected/3ade3fec-8e2a-4840-9390-43d3316df678-kube-api-access-9nkv9\") pod \"e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:18.963431 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:18.963402 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:19.092190 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:19.092127 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:19.254967 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:19.254932 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx"] Jul 07 18:52:19.257123 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:52:19.257086 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ade3fec_8e2a_4840_9390_43d3316df678.slice/crio-c52b906cff4f08856387c947b3445d978b1843fc4da520e62b57e547a123ceea WatchSource:0}: Error finding container c52b906cff4f08856387c947b3445d978b1843fc4da520e62b57e547a123ceea: Status 404 returned error can't find the container with id c52b906cff4f08856387c947b3445d978b1843fc4da520e62b57e547a123ceea Jul 07 18:52:19.259223 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:19.259185 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:52:19.259301 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:19.259254 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:52:19.259301 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:19.259280 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:52:19.630559 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:19.630520 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" event={"ID":"3ade3fec-8e2a-4840-9390-43d3316df678","Type":"ContainerStarted","Data":"afff8acfdca957f4f67dfac4cb0ac5580b217496800ba97bf3dc54f9e7affb31"} Jul 07 18:52:19.630725 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:19.630586 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" event={"ID":"3ade3fec-8e2a-4840-9390-43d3316df678","Type":"ContainerStarted","Data":"c52b906cff4f08856387c947b3445d978b1843fc4da520e62b57e547a123ceea"} Jul 07 18:52:19.662448 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:19.662398 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" podStartSLOduration=1.6623833129999999 podStartE2EDuration="1.662383313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:52:19.660072471 +0000 UTC m=+1763.043865876" watchObservedRunningTime="2026-07-07 18:52:19.662383313 +0000 UTC m=+1763.046176707" Jul 07 18:52:20.092498 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:20.092468 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:20.097144 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:20.097121 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:20.635187 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:20.635162 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:20.636156 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:20.636139 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:21.880805 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:21.880772 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:22.062341 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:22.062295 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:22.320408 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:22.320381 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:23.108722 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:23.108693 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:23.303639 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:23.303606 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:26.429026 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:26.428995 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:27.193963 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:27.193917 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:27.282756 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:27.282726 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:52:27.351105 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:27.351076 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:27.675256 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:27.675225 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:34.723529 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:34.723488 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:34.838776 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:34.838744 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:35.409533 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:35.409500 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:35.636789 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:35.636758 2578 patch_prober.go:28] interesting pod/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 07 18:52:35.636959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:35.636821 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" podUID="3ade3fec-8e2a-4840-9390-43d3316df678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 07 18:52:35.709697 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:35.709618 2578 generic.go:356] "Generic (PLEG): container finished" podID="3ade3fec-8e2a-4840-9390-43d3316df678" containerID="afff8acfdca957f4f67dfac4cb0ac5580b217496800ba97bf3dc54f9e7affb31" exitCode=0 Jul 07 18:52:35.709697 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:35.709683 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" event={"ID":"3ade3fec-8e2a-4840-9390-43d3316df678","Type":"ContainerDied","Data":"afff8acfdca957f4f67dfac4cb0ac5580b217496800ba97bf3dc54f9e7affb31"} Jul 07 18:52:35.710070 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:35.710052 2578 scope.go:117] "RemoveContainer" containerID="afff8acfdca957f4f67dfac4cb0ac5580b217496800ba97bf3dc54f9e7affb31" Jul 07 18:52:35.711003 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:35.710974 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:52:35.711061 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:35.711045 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:52:35.711097 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:35.711077 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:52:35.867283 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:35.867248 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:35.993676 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:35.993642 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:36.716584 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:36.716548 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" event={"ID":"3ade3fec-8e2a-4840-9390-43d3316df678","Type":"ContainerStarted","Data":"67939a0233d773fe6bc25058d77c1a23d777895598c14e5ab7f2acc9c109b72d"} Jul 07 18:52:37.092815 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:37.092774 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:37.093295 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:37.092900 2578 patch_prober.go:28] interesting pod/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 07 18:52:37.093295 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:37.092941 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" podUID="3ade3fec-8e2a-4840-9390-43d3316df678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 07 18:52:38.093585 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:38.093542 2578 patch_prober.go:28] interesting pod/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 07 18:52:38.094068 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:38.093607 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" podUID="3ade3fec-8e2a-4840-9390-43d3316df678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 07 18:52:39.093474 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:39.093437 2578 patch_prober.go:28] interesting pod/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 07 18:52:39.093762 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:39.093736 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" podUID="3ade3fec-8e2a-4840-9390-43d3316df678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 07 18:52:40.093501 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:40.093457 2578 patch_prober.go:28] interesting pod/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 07 18:52:40.093718 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:40.093523 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" podUID="3ade3fec-8e2a-4840-9390-43d3316df678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 07 18:52:40.764244 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:40.764212 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:52:41.097024 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:41.096953 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:41.097342 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:41.097306 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:41.098125 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:41.098104 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:43.381013 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:43.380982 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:43.523520 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:43.523488 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:43.745490 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:43.745450 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx"] Jul 07 18:52:43.750764 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:43.750731 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" podUID="3ade3fec-8e2a-4840-9390-43d3316df678" containerName="istio-proxy" containerID="cri-o://67939a0233d773fe6bc25058d77c1a23d777895598c14e5ab7f2acc9c109b72d" gracePeriod=30 Jul 07 18:52:43.798354 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:43.798307 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:44.015656 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:44.015585 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:44.187944 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:44.187906 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:44.382512 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:44.382443 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:44.590129 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:44.590105 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:49.005229 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.005199 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:49.172375 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.172263 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-workload-certs\") pod \"3ade3fec-8e2a-4840-9390-43d3316df678\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " Jul 07 18:52:49.172375 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.172299 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ade3fec-8e2a-4840-9390-43d3316df678-istio-podinfo\") pod \"3ade3fec-8e2a-4840-9390-43d3316df678\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " Jul 07 18:52:49.172375 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.172358 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ade3fec-8e2a-4840-9390-43d3316df678-istio-token\") pod \"3ade3fec-8e2a-4840-9390-43d3316df678\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " Jul 07 18:52:49.172375 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.172375 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-workload-socket\") pod \"3ade3fec-8e2a-4840-9390-43d3316df678\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " Jul 07 18:52:49.172713 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.172393 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-credential-socket\") pod \"3ade3fec-8e2a-4840-9390-43d3316df678\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " Jul 07 18:52:49.172713 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.172434 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-istio-envoy\") pod \"3ade3fec-8e2a-4840-9390-43d3316df678\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " Jul 07 18:52:49.172713 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.172530 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-istio-data\") pod \"3ade3fec-8e2a-4840-9390-43d3316df678\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " Jul 07 18:52:49.172713 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.172582 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nkv9\" (UniqueName: \"kubernetes.io/projected/3ade3fec-8e2a-4840-9390-43d3316df678-kube-api-access-9nkv9\") pod \"3ade3fec-8e2a-4840-9390-43d3316df678\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " Jul 07 18:52:49.172713 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.172581 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3ade3fec-8e2a-4840-9390-43d3316df678" (UID: "3ade3fec-8e2a-4840-9390-43d3316df678"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:52:49.172713 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.172614 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ade3fec-8e2a-4840-9390-43d3316df678-istiod-ca-cert\") pod \"3ade3fec-8e2a-4840-9390-43d3316df678\" (UID: \"3ade3fec-8e2a-4840-9390-43d3316df678\") " Jul 07 18:52:49.173019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.172735 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3ade3fec-8e2a-4840-9390-43d3316df678" (UID: "3ade3fec-8e2a-4840-9390-43d3316df678"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:52:49.173019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.172751 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3ade3fec-8e2a-4840-9390-43d3316df678" (UID: "3ade3fec-8e2a-4840-9390-43d3316df678"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:52:49.173019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.172945 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-workload-socket\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:52:49.173019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.172965 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-credential-socket\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:52:49.173019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.172980 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-workload-certs\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:52:49.173253 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.173049 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ade3fec-8e2a-4840-9390-43d3316df678-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3ade3fec-8e2a-4840-9390-43d3316df678" (UID: "3ade3fec-8e2a-4840-9390-43d3316df678"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:52:49.173582 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.173561 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-istio-data" (OuterVolumeSpecName: "istio-data") pod "3ade3fec-8e2a-4840-9390-43d3316df678" (UID: "3ade3fec-8e2a-4840-9390-43d3316df678"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:52:49.175152 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.175129 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3ade3fec-8e2a-4840-9390-43d3316df678" (UID: "3ade3fec-8e2a-4840-9390-43d3316df678"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:52:49.175236 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.175173 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ade3fec-8e2a-4840-9390-43d3316df678-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3ade3fec-8e2a-4840-9390-43d3316df678" (UID: "3ade3fec-8e2a-4840-9390-43d3316df678"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:52:49.175236 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.175174 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ade3fec-8e2a-4840-9390-43d3316df678-istio-token" (OuterVolumeSpecName: "istio-token") pod "3ade3fec-8e2a-4840-9390-43d3316df678" (UID: "3ade3fec-8e2a-4840-9390-43d3316df678"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:52:49.175236 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.175192 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ade3fec-8e2a-4840-9390-43d3316df678-kube-api-access-9nkv9" (OuterVolumeSpecName: "kube-api-access-9nkv9") pod "3ade3fec-8e2a-4840-9390-43d3316df678" (UID: "3ade3fec-8e2a-4840-9390-43d3316df678"). InnerVolumeSpecName "kube-api-access-9nkv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:52:49.274313 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.274291 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ade3fec-8e2a-4840-9390-43d3316df678-istio-podinfo\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:52:49.274313 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.274313 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ade3fec-8e2a-4840-9390-43d3316df678-istio-token\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:52:49.274465 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.274347 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-istio-envoy\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:52:49.274465 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.274359 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ade3fec-8e2a-4840-9390-43d3316df678-istio-data\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:52:49.274465 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.274371 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nkv9\" (UniqueName: \"kubernetes.io/projected/3ade3fec-8e2a-4840-9390-43d3316df678-kube-api-access-9nkv9\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:52:49.274465 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.274383 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ade3fec-8e2a-4840-9390-43d3316df678-istiod-ca-cert\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:52:49.778718 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.778688 2578 generic.go:356] "Generic (PLEG): container finished" podID="3ade3fec-8e2a-4840-9390-43d3316df678" containerID="67939a0233d773fe6bc25058d77c1a23d777895598c14e5ab7f2acc9c109b72d" exitCode=0 Jul 07 18:52:49.778888 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.778737 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" Jul 07 18:52:49.778888 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.778761 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" event={"ID":"3ade3fec-8e2a-4840-9390-43d3316df678","Type":"ContainerDied","Data":"67939a0233d773fe6bc25058d77c1a23d777895598c14e5ab7f2acc9c109b72d"} Jul 07 18:52:49.778888 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.778801 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx" event={"ID":"3ade3fec-8e2a-4840-9390-43d3316df678","Type":"ContainerDied","Data":"c52b906cff4f08856387c947b3445d978b1843fc4da520e62b57e547a123ceea"} Jul 07 18:52:49.778888 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.778822 2578 scope.go:117] "RemoveContainer" containerID="67939a0233d773fe6bc25058d77c1a23d777895598c14e5ab7f2acc9c109b72d" Jul 07 18:52:49.787258 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.787242 2578 scope.go:117] "RemoveContainer" containerID="afff8acfdca957f4f67dfac4cb0ac5580b217496800ba97bf3dc54f9e7affb31" Jul 07 18:52:49.794247 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.794212 2578 scope.go:117] "RemoveContainer" containerID="67939a0233d773fe6bc25058d77c1a23d777895598c14e5ab7f2acc9c109b72d" Jul 07 18:52:49.794571 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:52:49.794549 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67939a0233d773fe6bc25058d77c1a23d777895598c14e5ab7f2acc9c109b72d\": container with ID starting with 67939a0233d773fe6bc25058d77c1a23d777895598c14e5ab7f2acc9c109b72d not found: ID does not exist" containerID="67939a0233d773fe6bc25058d77c1a23d777895598c14e5ab7f2acc9c109b72d" Jul 07 18:52:49.794678 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.794583 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67939a0233d773fe6bc25058d77c1a23d777895598c14e5ab7f2acc9c109b72d"} err="failed to get container status \"67939a0233d773fe6bc25058d77c1a23d777895598c14e5ab7f2acc9c109b72d\": rpc error: code = NotFound desc = could not find container \"67939a0233d773fe6bc25058d77c1a23d777895598c14e5ab7f2acc9c109b72d\": container with ID starting with 67939a0233d773fe6bc25058d77c1a23d777895598c14e5ab7f2acc9c109b72d not found: ID does not exist" Jul 07 18:52:49.794678 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.794607 2578 scope.go:117] "RemoveContainer" containerID="afff8acfdca957f4f67dfac4cb0ac5580b217496800ba97bf3dc54f9e7affb31" Jul 07 18:52:49.794963 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:52:49.794943 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afff8acfdca957f4f67dfac4cb0ac5580b217496800ba97bf3dc54f9e7affb31\": container with ID starting with afff8acfdca957f4f67dfac4cb0ac5580b217496800ba97bf3dc54f9e7affb31 not found: ID does not exist" containerID="afff8acfdca957f4f67dfac4cb0ac5580b217496800ba97bf3dc54f9e7affb31" Jul 07 18:52:49.795074 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.794971 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afff8acfdca957f4f67dfac4cb0ac5580b217496800ba97bf3dc54f9e7affb31"} err="failed to get container status \"afff8acfdca957f4f67dfac4cb0ac5580b217496800ba97bf3dc54f9e7affb31\": rpc error: code = NotFound desc = could not find container \"afff8acfdca957f4f67dfac4cb0ac5580b217496800ba97bf3dc54f9e7affb31\": container with ID starting with afff8acfdca957f4f67dfac4cb0ac5580b217496800ba97bf3dc54f9e7affb31 not found: ID does not exist" Jul 07 18:52:49.796460 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.796443 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx"] Jul 07 18:52:49.801435 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:49.801414 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8c3df14f-openshift-default-5dc88d5d5f-w7bkx"] Jul 07 18:52:50.567397 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:50.567363 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:50.711077 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:50.711047 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:50.932249 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:50.932171 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:51.030817 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:51.030795 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:51.196652 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:51.196582 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ade3fec-8e2a-4840-9390-43d3316df678" path="/var/lib/kubelet/pods/3ade3fec-8e2a-4840-9390-43d3316df678/volumes" Jul 07 18:52:57.164643 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:57.164616 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 18:52:57.167895 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:57.167876 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 18:52:57.448588 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:57.448520 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:57.544545 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:57.544490 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:58.736558 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:58.736523 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:52:58.856984 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:52:58.856944 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:53:06.990694 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:06.990662 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:53:07.338175 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.338095 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf"] Jul 07 18:53:07.338464 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.338449 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ade3fec-8e2a-4840-9390-43d3316df678" containerName="istio-proxy" Jul 07 18:53:07.338515 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.338467 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ade3fec-8e2a-4840-9390-43d3316df678" containerName="istio-proxy" Jul 07 18:53:07.338551 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.338532 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ade3fec-8e2a-4840-9390-43d3316df678" containerName="istio-proxy" Jul 07 18:53:07.338551 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.338546 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ade3fec-8e2a-4840-9390-43d3316df678" containerName="istio-proxy" Jul 07 18:53:07.342697 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.342680 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.345983 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.345960 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3ae90c5b-openshift-default-dockercfg-nlcqw\"" Jul 07 18:53:07.357451 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.357428 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf"] Jul 07 18:53:07.410719 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.410691 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-workload-socket\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.410869 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.410726 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-envoy\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.410869 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.410762 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-credential-socket\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.410869 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.410848 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-token\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.411028 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.410898 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istiod-ca-cert\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.411028 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.410945 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-podinfo\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.411104 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.411013 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-workload-certs\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.411104 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.411064 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95kn4\" (UniqueName: \"kubernetes.io/projected/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-kube-api-access-95kn4\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.411104 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.411096 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-data\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.511569 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.511532 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95kn4\" (UniqueName: \"kubernetes.io/projected/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-kube-api-access-95kn4\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.511727 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.511573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-data\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.511727 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.511605 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-workload-socket\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.511727 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.511628 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-envoy\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.511727 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.511652 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-credential-socket\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.511727 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.511678 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-token\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.512054 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.512028 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istiod-ca-cert\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.512132 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.512083 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-podinfo\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.512189 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.512129 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-workload-socket\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.512189 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.512140 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-workload-certs\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.512189 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.512150 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-data\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.512434 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.512414 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-credential-socket\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.512508 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.512488 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-workload-certs\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.512885 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.512857 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istiod-ca-cert\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.514608 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.514586 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-envoy\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.514714 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.514651 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-podinfo\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.521439 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.521413 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95kn4\" (UniqueName: \"kubernetes.io/projected/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-kube-api-access-95kn4\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.521791 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.521768 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-token\") pod \"e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.617343 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.617252 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:53:07.653310 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.653285 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:07.809204 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:53:07.809171 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ec7105_8e72_4f6b_8bc3_59add6a5b5e6.slice/crio-c375b88391e8e7c2221dfa469c10038bade8026cbed1fad30e7dbbb5e7286c1c WatchSource:0}: Error finding container c375b88391e8e7c2221dfa469c10038bade8026cbed1fad30e7dbbb5e7286c1c: Status 404 returned error can't find the container with id c375b88391e8e7c2221dfa469c10038bade8026cbed1fad30e7dbbb5e7286c1c Jul 07 18:53:07.809891 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.809866 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf"] Jul 07 18:53:07.811485 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.811457 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:53:07.811578 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.811528 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:53:07.811578 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.811571 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:53:07.858807 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.858780 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" event={"ID":"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6","Type":"ContainerStarted","Data":"c375b88391e8e7c2221dfa469c10038bade8026cbed1fad30e7dbbb5e7286c1c"} Jul 07 18:53:07.973001 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:07.972970 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:53:08.864797 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:08.864762 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" event={"ID":"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6","Type":"ContainerStarted","Data":"01e52661127f959183243c479b748fa7de0ff8ed1acb28d9fdbc4290de27e432"} Jul 07 18:53:08.900366 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:08.900307 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" podStartSLOduration=1.9002933309999999 podStartE2EDuration="1.900293331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:53:08.896939822 +0000 UTC m=+1812.280733228" watchObservedRunningTime="2026-07-07 18:53:08.900293331 +0000 UTC m=+1812.284086726" Jul 07 18:53:09.654065 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:09.654031 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:09.658731 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:09.658707 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:09.869482 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:09.869454 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:09.870311 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:09.870292 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:10.984264 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:10.984233 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:53:11.079716 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:11.079684 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:53:11.222857 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:11.222824 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:53:11.450796 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:11.450720 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:53:11.879880 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:11.879844 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:53:11.980106 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:11.980070 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:53:15.506445 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:15.506408 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:53:17.089869 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:17.089838 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:53:24.871029 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:24.870990 2578 patch_prober.go:28] interesting pod/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 07 18:53:24.871471 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:24.871059 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" podUID="c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 07 18:53:24.938634 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:24.938605 2578 generic.go:356] "Generic (PLEG): container finished" podID="c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" containerID="01e52661127f959183243c479b748fa7de0ff8ed1acb28d9fdbc4290de27e432" exitCode=0 Jul 07 18:53:24.938780 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:24.938673 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" event={"ID":"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6","Type":"ContainerDied","Data":"01e52661127f959183243c479b748fa7de0ff8ed1acb28d9fdbc4290de27e432"} Jul 07 18:53:24.939064 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:24.939049 2578 scope.go:117] "RemoveContainer" containerID="01e52661127f959183243c479b748fa7de0ff8ed1acb28d9fdbc4290de27e432" Jul 07 18:53:24.940077 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:24.940048 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:53:24.940150 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:24.940132 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:53:24.940203 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:24.940189 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:53:25.945566 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:25.945531 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" event={"ID":"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6","Type":"ContainerStarted","Data":"732d3aaa2a15feecae7003a734a505906b55b274824002bab03687d01e8a8fac"} Jul 07 18:53:26.654460 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:26.654426 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:26.654636 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:26.654580 2578 patch_prober.go:28] interesting pod/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 07 18:53:26.654636 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:26.654623 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" podUID="c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 07 18:53:27.621556 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.621530 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7"] Jul 07 18:53:27.621910 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.621844 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ade3fec-8e2a-4840-9390-43d3316df678" containerName="istio-proxy" Jul 07 18:53:27.621910 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.621855 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ade3fec-8e2a-4840-9390-43d3316df678" containerName="istio-proxy" Jul 07 18:53:27.624655 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.624638 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:27.630440 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.630417 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3ae90c5b\"/\"default-dockercfg-dnzrd\"" Jul 07 18:53:27.631099 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.630960 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3ae90c5b\"/\"kube-root-ca.crt\"" Jul 07 18:53:27.631608 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.631586 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3ae90c5b\"/\"openshift-service-ca.crt\"" Jul 07 18:53:27.633057 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.633039 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3ae90c5b\"/\"e2e-lifecycle-model-3ae90c5b-kserve-self-signed-certs\"" Jul 07 18:53:27.641350 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.641296 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7"] Jul 07 18:53:27.654606 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.654578 2578 patch_prober.go:28] interesting pod/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 07 18:53:27.654711 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.654629 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" podUID="c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 07 18:53:27.686341 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.685862 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:53:27.686482 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.686356 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-model-cache\") pod \"e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:27.686482 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.686400 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-dshm\") pod \"e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:27.686482 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.686424 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9570f0-2032-4cc6-87ad-1e57459826eb-tls-certs\") pod \"e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:27.686660 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.686505 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-home\") pod \"e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:27.686660 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.686556 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-tmp-dir\") pod \"e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:27.686660 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.686582 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjnt9\" (UniqueName: \"kubernetes.io/projected/bb9570f0-2032-4cc6-87ad-1e57459826eb-kube-api-access-sjnt9\") pod \"e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:27.788036 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.787998 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-tmp-dir\") pod \"e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:27.788232 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.788047 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjnt9\" (UniqueName: \"kubernetes.io/projected/bb9570f0-2032-4cc6-87ad-1e57459826eb-kube-api-access-sjnt9\") pod \"e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:27.788232 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.788080 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-model-cache\") pod \"e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:27.788232 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.788121 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-dshm\") pod \"e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:27.788232 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.788145 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9570f0-2032-4cc6-87ad-1e57459826eb-tls-certs\") pod \"e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:27.788566 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.788235 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-home\") pod \"e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:27.788708 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.788684 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-home\") pod \"e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:27.788768 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.788734 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-model-cache\") pod \"e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:27.789056 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.789026 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-tmp-dir\") pod \"e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:27.793648 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.793615 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-dshm\") pod \"e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:27.799876 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.799852 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9570f0-2032-4cc6-87ad-1e57459826eb-tls-certs\") pod \"e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:27.806256 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.806232 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjnt9\" (UniqueName: \"kubernetes.io/projected/bb9570f0-2032-4cc6-87ad-1e57459826eb-kube-api-access-sjnt9\") pod \"e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:27.934347 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:27.934252 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:28.117534 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:28.117509 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7"] Jul 07 18:53:28.119730 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:53:28.119700 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb9570f0_2032_4cc6_87ad_1e57459826eb.slice/crio-78ac67d5e6b3a492480c689bb6be8f6dd3baf07838f8bf74e486bc8394dcadd8 WatchSource:0}: Error finding container 78ac67d5e6b3a492480c689bb6be8f6dd3baf07838f8bf74e486bc8394dcadd8: Status 404 returned error can't find the container with id 78ac67d5e6b3a492480c689bb6be8f6dd3baf07838f8bf74e486bc8394dcadd8 Jul 07 18:53:28.654457 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:28.654422 2578 patch_prober.go:28] interesting pod/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 07 18:53:28.654907 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:28.654479 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" podUID="c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 07 18:53:28.960723 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:28.960628 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" event={"ID":"bb9570f0-2032-4cc6-87ad-1e57459826eb","Type":"ContainerStarted","Data":"4652a9b6e0eda9226062330ad6467ee511b49b4a22397e16d86920e4c11e1ba8"} Jul 07 18:53:28.960723 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:28.960682 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" event={"ID":"bb9570f0-2032-4cc6-87ad-1e57459826eb","Type":"ContainerStarted","Data":"78ac67d5e6b3a492480c689bb6be8f6dd3baf07838f8bf74e486bc8394dcadd8"} Jul 07 18:53:28.984265 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:28.984201 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" podStartSLOduration=1.9841811520000001 podStartE2EDuration="1.984181152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:53:28.981018734 +0000 UTC m=+1832.364812125" watchObservedRunningTime="2026-07-07 18:53:28.984181152 +0000 UTC m=+1832.367974549" Jul 07 18:53:29.653820 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:29.653783 2578 patch_prober.go:28] interesting pod/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 07 18:53:29.654006 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:29.653838 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" podUID="c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 07 18:53:30.657530 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:30.657501 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:30.657899 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:30.657746 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:30.658400 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:30.658384 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:53:31.687843 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:31.687806 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:53:37.935068 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:37.935028 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:37.935482 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:37.935084 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:37.947225 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:37.947203 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:38.016054 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:38.016030 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:53:45.756311 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:45.756278 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:53:48.726620 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:48.726581 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:53:49.636125 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:49.636087 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:53:54.947158 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:54.947126 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:53:56.050167 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:56.050136 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:53:59.877407 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:59.877375 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7"] Jul 07 18:53:59.959452 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:53:59.959422 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:54:04.890688 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:04.890650 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:54:14.897118 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:14.897078 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" podUID="bb9570f0-2032-4cc6-87ad-1e57459826eb" containerName="main" containerID="cri-o://4652a9b6e0eda9226062330ad6467ee511b49b4a22397e16d86920e4c11e1ba8" gracePeriod=45 Jul 07 18:54:15.154309 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.154253 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:54:15.173608 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.173583 2578 generic.go:356] "Generic (PLEG): container finished" podID="bb9570f0-2032-4cc6-87ad-1e57459826eb" containerID="4652a9b6e0eda9226062330ad6467ee511b49b4a22397e16d86920e4c11e1ba8" exitCode=0 Jul 07 18:54:15.173728 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.173619 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" Jul 07 18:54:15.173728 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.173673 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" event={"ID":"bb9570f0-2032-4cc6-87ad-1e57459826eb","Type":"ContainerDied","Data":"4652a9b6e0eda9226062330ad6467ee511b49b4a22397e16d86920e4c11e1ba8"} Jul 07 18:54:15.173728 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.173714 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7" event={"ID":"bb9570f0-2032-4cc6-87ad-1e57459826eb","Type":"ContainerDied","Data":"78ac67d5e6b3a492480c689bb6be8f6dd3baf07838f8bf74e486bc8394dcadd8"} Jul 07 18:54:15.173844 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.173738 2578 scope.go:117] "RemoveContainer" containerID="4652a9b6e0eda9226062330ad6467ee511b49b4a22397e16d86920e4c11e1ba8" Jul 07 18:54:15.183222 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.183202 2578 scope.go:117] "RemoveContainer" containerID="4652a9b6e0eda9226062330ad6467ee511b49b4a22397e16d86920e4c11e1ba8" Jul 07 18:54:15.183519 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:54:15.183501 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4652a9b6e0eda9226062330ad6467ee511b49b4a22397e16d86920e4c11e1ba8\": container with ID starting with 4652a9b6e0eda9226062330ad6467ee511b49b4a22397e16d86920e4c11e1ba8 not found: ID does not exist" containerID="4652a9b6e0eda9226062330ad6467ee511b49b4a22397e16d86920e4c11e1ba8" Jul 07 18:54:15.183575 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.183534 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4652a9b6e0eda9226062330ad6467ee511b49b4a22397e16d86920e4c11e1ba8"} err="failed to get container status \"4652a9b6e0eda9226062330ad6467ee511b49b4a22397e16d86920e4c11e1ba8\": rpc error: code = NotFound desc = could not find container \"4652a9b6e0eda9226062330ad6467ee511b49b4a22397e16d86920e4c11e1ba8\": container with ID starting with 4652a9b6e0eda9226062330ad6467ee511b49b4a22397e16d86920e4c11e1ba8 not found: ID does not exist" Jul 07 18:54:15.274560 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.274534 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-home\") pod \"bb9570f0-2032-4cc6-87ad-1e57459826eb\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " Jul 07 18:54:15.274699 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.274569 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9570f0-2032-4cc6-87ad-1e57459826eb-tls-certs\") pod \"bb9570f0-2032-4cc6-87ad-1e57459826eb\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " Jul 07 18:54:15.274699 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.274592 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-tmp-dir\") pod \"bb9570f0-2032-4cc6-87ad-1e57459826eb\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " Jul 07 18:54:15.274699 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.274625 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-dshm\") pod \"bb9570f0-2032-4cc6-87ad-1e57459826eb\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " Jul 07 18:54:15.274699 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.274683 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-model-cache\") pod \"bb9570f0-2032-4cc6-87ad-1e57459826eb\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " Jul 07 18:54:15.274918 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.274721 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjnt9\" (UniqueName: \"kubernetes.io/projected/bb9570f0-2032-4cc6-87ad-1e57459826eb-kube-api-access-sjnt9\") pod \"bb9570f0-2032-4cc6-87ad-1e57459826eb\" (UID: \"bb9570f0-2032-4cc6-87ad-1e57459826eb\") " Jul 07 18:54:15.274918 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.274830 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-home" (OuterVolumeSpecName: "home") pod "bb9570f0-2032-4cc6-87ad-1e57459826eb" (UID: "bb9570f0-2032-4cc6-87ad-1e57459826eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:54:15.274918 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.274855 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb9570f0-2032-4cc6-87ad-1e57459826eb" (UID: "bb9570f0-2032-4cc6-87ad-1e57459826eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:54:15.275059 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.274992 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb9570f0-2032-4cc6-87ad-1e57459826eb" (UID: "bb9570f0-2032-4cc6-87ad-1e57459826eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:54:15.275059 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.275037 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-tmp-dir\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:54:15.275059 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.275055 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-home\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:54:15.276639 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.276614 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb9570f0-2032-4cc6-87ad-1e57459826eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb9570f0-2032-4cc6-87ad-1e57459826eb" (UID: "bb9570f0-2032-4cc6-87ad-1e57459826eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:54:15.276727 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.276695 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-dshm" (OuterVolumeSpecName: "dshm") pod "bb9570f0-2032-4cc6-87ad-1e57459826eb" (UID: "bb9570f0-2032-4cc6-87ad-1e57459826eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:54:15.276771 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.276755 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9570f0-2032-4cc6-87ad-1e57459826eb-kube-api-access-sjnt9" (OuterVolumeSpecName: "kube-api-access-sjnt9") pod "bb9570f0-2032-4cc6-87ad-1e57459826eb" (UID: "bb9570f0-2032-4cc6-87ad-1e57459826eb"). InnerVolumeSpecName "kube-api-access-sjnt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:54:15.376459 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.376437 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-model-cache\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:54:15.376459 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.376458 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjnt9\" (UniqueName: \"kubernetes.io/projected/bb9570f0-2032-4cc6-87ad-1e57459826eb-kube-api-access-sjnt9\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:54:15.376614 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.376468 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9570f0-2032-4cc6-87ad-1e57459826eb-tls-certs\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:54:15.376614 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.376477 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9570f0-2032-4cc6-87ad-1e57459826eb-dshm\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:54:15.499950 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.499926 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7"] Jul 07 18:54:15.506871 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:15.506853 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3ae90c5b/e2e-lifecycle-model-3ae90c5b-kserve-85cd6cff95-jhlp7"] Jul 07 18:54:17.201457 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:17.201385 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb9570f0-2032-4cc6-87ad-1e57459826eb" path="/var/lib/kubelet/pods/bb9570f0-2032-4cc6-87ad-1e57459826eb/volumes" Jul 07 18:54:21.607444 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:21.607398 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:54:21.684508 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:21.684461 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:54:38.244913 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:38.244881 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:54:38.433111 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:38.433078 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:54:39.655304 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:39.655270 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:54:44.673150 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:54:44.673118 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:55:09.173928 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:09.173889 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:55:09.323736 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:09.323707 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:55:09.578712 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:09.578680 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf"] Jul 07 18:55:09.578905 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:09.578881 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" podUID="c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" containerName="istio-proxy" containerID="cri-o://732d3aaa2a15feecae7003a734a505906b55b274824002bab03687d01e8a8fac" gracePeriod=30 Jul 07 18:55:09.605121 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:09.605092 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:55:09.813088 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:09.813057 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:55:11.015573 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:11.015529 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:55:11.137548 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:11.137508 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:55:11.253255 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:11.253223 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:55:12.074873 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:12.074840 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:55:12.173761 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:12.173721 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:55:12.277967 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:12.277929 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:55:13.433099 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:13.433067 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:55:13.954057 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:13.954012 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:55:14.833450 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.833425 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:55:14.931665 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.931598 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-credential-socket\") pod \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " Jul 07 18:55:14.931665 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.931633 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-workload-certs\") pod \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " Jul 07 18:55:14.931851 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.931675 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-workload-socket\") pod \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " Jul 07 18:55:14.931851 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.931697 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-envoy\") pod \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " Jul 07 18:55:14.931851 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.931715 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istiod-ca-cert\") pod \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " Jul 07 18:55:14.931851 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.931734 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95kn4\" (UniqueName: \"kubernetes.io/projected/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-kube-api-access-95kn4\") pod \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " Jul 07 18:55:14.931851 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.931781 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-podinfo\") pod \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " Jul 07 18:55:14.932113 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.931873 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" (UID: "c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:55:14.932113 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.931903 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-data\") pod \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " Jul 07 18:55:14.932113 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.931942 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-token\") pod \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\" (UID: \"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6\") " Jul 07 18:55:14.932113 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.932004 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" (UID: "c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:55:14.932356 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.932158 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" (UID: "c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:55:14.932356 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.932251 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-credential-socket\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:55:14.932356 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.932268 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-workload-socket\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:55:14.932356 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.932282 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istiod-ca-cert\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:55:14.932564 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.932432 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" (UID: "c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:55:14.932712 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.932690 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-data" (OuterVolumeSpecName: "istio-data") pod "c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" (UID: "c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:55:14.933868 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.933840 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" (UID: "c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:55:14.933991 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.933898 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-kube-api-access-95kn4" (OuterVolumeSpecName: "kube-api-access-95kn4") pod "c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" (UID: "c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6"). InnerVolumeSpecName "kube-api-access-95kn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:55:14.933991 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.933952 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" (UID: "c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:55:14.933991 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:14.933987 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-token" (OuterVolumeSpecName: "istio-token") pod "c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" (UID: "c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:55:15.032991 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:15.032965 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-workload-certs\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:55:15.032991 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:15.032986 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-envoy\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:55:15.033120 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:15.032995 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95kn4\" (UniqueName: \"kubernetes.io/projected/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-kube-api-access-95kn4\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:55:15.033120 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:15.033005 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-podinfo\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:55:15.033120 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:15.033014 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-data\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:55:15.033120 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:15.033023 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6-istio-token\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:55:15.446549 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:15.446524 2578 generic.go:356] "Generic (PLEG): container finished" podID="c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" containerID="732d3aaa2a15feecae7003a734a505906b55b274824002bab03687d01e8a8fac" exitCode=0 Jul 07 18:55:15.446685 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:15.446574 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" Jul 07 18:55:15.446685 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:15.446607 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" event={"ID":"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6","Type":"ContainerDied","Data":"732d3aaa2a15feecae7003a734a505906b55b274824002bab03687d01e8a8fac"} Jul 07 18:55:15.446685 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:15.446659 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf" event={"ID":"c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6","Type":"ContainerDied","Data":"c375b88391e8e7c2221dfa469c10038bade8026cbed1fad30e7dbbb5e7286c1c"} Jul 07 18:55:15.446685 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:15.446683 2578 scope.go:117] "RemoveContainer" containerID="732d3aaa2a15feecae7003a734a505906b55b274824002bab03687d01e8a8fac" Jul 07 18:55:15.455055 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:15.455014 2578 scope.go:117] "RemoveContainer" containerID="01e52661127f959183243c479b748fa7de0ff8ed1acb28d9fdbc4290de27e432" Jul 07 18:55:15.463788 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:15.463771 2578 scope.go:117] "RemoveContainer" containerID="732d3aaa2a15feecae7003a734a505906b55b274824002bab03687d01e8a8fac" Jul 07 18:55:15.464015 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:55:15.463997 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"732d3aaa2a15feecae7003a734a505906b55b274824002bab03687d01e8a8fac\": container with ID starting with 732d3aaa2a15feecae7003a734a505906b55b274824002bab03687d01e8a8fac not found: ID does not exist" containerID="732d3aaa2a15feecae7003a734a505906b55b274824002bab03687d01e8a8fac" Jul 07 18:55:15.464073 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:15.464021 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732d3aaa2a15feecae7003a734a505906b55b274824002bab03687d01e8a8fac"} err="failed to get container status \"732d3aaa2a15feecae7003a734a505906b55b274824002bab03687d01e8a8fac\": rpc error: code = NotFound desc = could not find container \"732d3aaa2a15feecae7003a734a505906b55b274824002bab03687d01e8a8fac\": container with ID starting with 732d3aaa2a15feecae7003a734a505906b55b274824002bab03687d01e8a8fac not found: ID does not exist" Jul 07 18:55:15.464073 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:15.464037 2578 scope.go:117] "RemoveContainer" containerID="01e52661127f959183243c479b748fa7de0ff8ed1acb28d9fdbc4290de27e432" Jul 07 18:55:15.464271 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:55:15.464249 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01e52661127f959183243c479b748fa7de0ff8ed1acb28d9fdbc4290de27e432\": container with ID starting with 01e52661127f959183243c479b748fa7de0ff8ed1acb28d9fdbc4290de27e432 not found: ID does not exist" containerID="01e52661127f959183243c479b748fa7de0ff8ed1acb28d9fdbc4290de27e432" Jul 07 18:55:15.464383 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:15.464268 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01e52661127f959183243c479b748fa7de0ff8ed1acb28d9fdbc4290de27e432"} err="failed to get container status \"01e52661127f959183243c479b748fa7de0ff8ed1acb28d9fdbc4290de27e432\": rpc error: code = NotFound desc = could not find container \"01e52661127f959183243c479b748fa7de0ff8ed1acb28d9fdbc4290de27e432\": container with ID starting with 01e52661127f959183243c479b748fa7de0ff8ed1acb28d9fdbc4290de27e432 not found: ID does not exist" Jul 07 18:55:15.467015 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:15.466994 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf"] Jul 07 18:55:15.474297 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:15.474268 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3ae90c5b-openshift-default-8666c95b7-nptgf"] Jul 07 18:55:17.196826 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:17.196793 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" path="/var/lib/kubelet/pods/c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6/volumes" Jul 07 18:55:20.652272 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:20.652226 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:55:20.873183 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:20.873130 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:55:21.391172 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:21.391146 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:55:25.178599 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:25.178559 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:55:27.688392 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:27.688358 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:55:29.067814 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:29.067720 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:55:38.500399 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:38.500363 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:55:38.944730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:55:38.944618 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:00.470595 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:00.470561 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:06.859729 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:06.859693 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:06.961594 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:06.961568 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:07.461290 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:07.461259 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:08.009101 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.009073 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw"] Jul 07 18:56:08.009477 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.009415 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" containerName="istio-proxy" Jul 07 18:56:08.009477 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.009427 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" containerName="istio-proxy" Jul 07 18:56:08.009477 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.009448 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb9570f0-2032-4cc6-87ad-1e57459826eb" containerName="main" Jul 07 18:56:08.009477 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.009453 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb9570f0-2032-4cc6-87ad-1e57459826eb" containerName="main" Jul 07 18:56:08.009609 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.009509 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" containerName="istio-proxy" Jul 07 18:56:08.009609 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.009519 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb9570f0-2032-4cc6-87ad-1e57459826eb" containerName="main" Jul 07 18:56:08.009609 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.009525 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" containerName="istio-proxy" Jul 07 18:56:08.012481 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.012462 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.022815 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.022793 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b07e06-openshift-default-dockercfg-gqs4h\"" Jul 07 18:56:08.048187 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.048128 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:08.068424 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.068401 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw"] Jul 07 18:56:08.116933 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.116907 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-podinfo\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.117053 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.116956 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-data\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.117053 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.117022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f667ca82-13e6-4f98-927c-c376fec7de6d-istiod-ca-cert\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.117128 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.117075 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-envoy\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.117128 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.117103 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7949\" (UniqueName: \"kubernetes.io/projected/f667ca82-13e6-4f98-927c-c376fec7de6d-kube-api-access-t7949\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.117193 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.117128 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-credential-socket\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.117193 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.117160 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-workload-socket\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.117193 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.117187 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-token\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.117288 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.117205 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-workload-certs\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.186794 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.186767 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:08.217935 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.217911 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-token\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.218045 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.217939 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-workload-certs\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.218045 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.217962 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-podinfo\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.218045 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.217988 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-data\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.218045 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.218022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f667ca82-13e6-4f98-927c-c376fec7de6d-istiod-ca-cert\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.218263 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.218056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-envoy\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.218263 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.218092 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7949\" (UniqueName: \"kubernetes.io/projected/f667ca82-13e6-4f98-927c-c376fec7de6d-kube-api-access-t7949\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.218263 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.218123 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-credential-socket\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.218263 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.218151 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-workload-socket\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.218496 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.218421 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-data\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.218552 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.218540 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-workload-socket\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.218610 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.218590 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-credential-socket\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.218665 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.218642 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-workload-certs\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.218951 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.218927 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f667ca82-13e6-4f98-927c-c376fec7de6d-istiod-ca-cert\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.220265 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.220244 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-envoy\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.220469 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.220453 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-podinfo\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.227813 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.227794 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-token\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.227972 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.227951 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7949\" (UniqueName: \"kubernetes.io/projected/f667ca82-13e6-4f98-927c-c376fec7de6d-kube-api-access-t7949\") pod \"e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.322168 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.322108 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:08.480349 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:56:08.480304 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf667ca82_13e6_4f98_927c_c376fec7de6d.slice/crio-12014ba313c8b5e80bad6688a85da4ff7122b3c3259e5cbe5779eb104b74396a WatchSource:0}: Error finding container 12014ba313c8b5e80bad6688a85da4ff7122b3c3259e5cbe5779eb104b74396a: Status 404 returned error can't find the container with id 12014ba313c8b5e80bad6688a85da4ff7122b3c3259e5cbe5779eb104b74396a Jul 07 18:56:08.480940 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.480918 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw"] Jul 07 18:56:08.482836 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.482804 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:56:08.482913 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.482869 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:56:08.482913 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.482898 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:56:08.668821 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.668726 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" event={"ID":"f667ca82-13e6-4f98-927c-c376fec7de6d","Type":"ContainerStarted","Data":"2934887cc2c4fb4f1b6d8c115aa1f67f6a218129d592abe6943aa6e45903e925"} Jul 07 18:56:08.668821 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.668794 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" event={"ID":"f667ca82-13e6-4f98-927c-c376fec7de6d","Type":"ContainerStarted","Data":"12014ba313c8b5e80bad6688a85da4ff7122b3c3259e5cbe5779eb104b74396a"} Jul 07 18:56:08.706554 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:08.706492 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" podStartSLOduration=1.706472258 podStartE2EDuration="1.706472258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:56:08.704692972 +0000 UTC m=+1992.088486376" watchObservedRunningTime="2026-07-07 18:56:08.706472258 +0000 UTC m=+1992.090265653" Jul 07 18:56:09.323159 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:09.323130 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:09.327867 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:09.327843 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:09.674209 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:09.674131 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:09.674920 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:09.674896 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:11.441420 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:11.441389 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:11.550997 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:11.550949 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:11.658113 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:11.658081 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:12.226168 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:12.226123 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:12.319961 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:12.319934 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:16.014691 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:16.014663 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:16.375884 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:16.375787 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:16.772432 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:16.772394 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:16.855624 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:16.855592 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:17.664795 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:17.664766 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:56:20.166936 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:20.166907 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:20.285197 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:20.285158 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:20.376541 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:20.376448 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:21.144786 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:21.144748 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:21.265496 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:21.265464 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:24.757885 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:24.757851 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:25.113470 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:25.112994 2578 patch_prober.go:28] interesting pod/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": read tcp 10.134.0.2:47328->10.134.0.46:15021: read: connection reset by peer" start-of-body= Jul 07 18:56:25.113470 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:25.113061 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" podUID="f667ca82-13e6-4f98-927c-c376fec7de6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": read tcp 10.134.0.2:47328->10.134.0.46:15021: read: connection reset by peer" Jul 07 18:56:25.745465 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:25.745434 2578 generic.go:356] "Generic (PLEG): container finished" podID="f667ca82-13e6-4f98-927c-c376fec7de6d" containerID="2934887cc2c4fb4f1b6d8c115aa1f67f6a218129d592abe6943aa6e45903e925" exitCode=0 Jul 07 18:56:25.745631 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:25.745507 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" event={"ID":"f667ca82-13e6-4f98-927c-c376fec7de6d","Type":"ContainerDied","Data":"2934887cc2c4fb4f1b6d8c115aa1f67f6a218129d592abe6943aa6e45903e925"} Jul 07 18:56:25.745851 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:25.745835 2578 scope.go:117] "RemoveContainer" containerID="2934887cc2c4fb4f1b6d8c115aa1f67f6a218129d592abe6943aa6e45903e925" Jul 07 18:56:25.746730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:25.746709 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:56:25.746779 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:25.746766 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:56:25.746817 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:25.746798 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:56:26.757067 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:26.757033 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" event={"ID":"f667ca82-13e6-4f98-927c-c376fec7de6d","Type":"ContainerStarted","Data":"bfc598764d718b5130e8bddf53012b71dc788fb42c7512010e39e229d12fffe0"} Jul 07 18:56:27.323264 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:27.323233 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:27.323464 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:27.323442 2578 patch_prober.go:28] interesting pod/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 07 18:56:27.323546 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:27.323498 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" podUID="f667ca82-13e6-4f98-927c-c376fec7de6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 07 18:56:28.323233 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:28.323198 2578 patch_prober.go:28] interesting pod/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 07 18:56:28.323696 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:28.323253 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" podUID="f667ca82-13e6-4f98-927c-c376fec7de6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 07 18:56:29.323230 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:29.323196 2578 patch_prober.go:28] interesting pod/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 07 18:56:29.323427 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:29.323245 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" podUID="f667ca82-13e6-4f98-927c-c376fec7de6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 07 18:56:30.333150 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:30.333120 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:30.333607 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:30.333424 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:30.334124 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:30.334107 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:56:31.495469 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:31.495434 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:56:34.239959 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:34.239924 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:34.346715 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:34.346681 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:41.997737 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:41.997705 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:42.093904 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:42.093872 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:51.245889 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.245849 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7"] Jul 07 18:56:51.246268 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.246208 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" containerName="istio-proxy" Jul 07 18:56:51.246268 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.246221 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2ec7105-8e72-4f6b-8bc3-59add6a5b5e6" containerName="istio-proxy" Jul 07 18:56:51.249191 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.249176 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:56:51.254075 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.254051 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b07e06\"/\"default-dockercfg-k54rj\"" Jul 07 18:56:51.254238 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.254051 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b07e06\"/\"test-model-b07e06-kserve-self-signed-certs\"" Jul 07 18:56:51.254238 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.254089 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b07e06\"/\"openshift-service-ca.crt\"" Jul 07 18:56:51.254238 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.254055 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b07e06\"/\"kube-root-ca.crt\"" Jul 07 18:56:51.261730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.261704 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7"] Jul 07 18:56:51.277260 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.277229 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:56:51.381394 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.381335 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-868v5\" (UniqueName: \"kubernetes.io/projected/91de2498-e273-4dea-a47b-b0914894c354-kube-api-access-868v5\") pod \"test-model-b07e06-kserve-5879c89f5c-6kks7\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:56:51.381587 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.381415 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-dshm\") pod \"test-model-b07e06-kserve-5879c89f5c-6kks7\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:56:51.381587 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.381452 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-tmp-dir\") pod \"test-model-b07e06-kserve-5879c89f5c-6kks7\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:56:51.381587 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.381487 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91de2498-e273-4dea-a47b-b0914894c354-tls-certs\") pod \"test-model-b07e06-kserve-5879c89f5c-6kks7\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:56:51.381587 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.381528 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-model-cache\") pod \"test-model-b07e06-kserve-5879c89f5c-6kks7\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:56:51.381820 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.381611 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-home\") pod \"test-model-b07e06-kserve-5879c89f5c-6kks7\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:56:51.482701 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.482662 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-home\") pod \"test-model-b07e06-kserve-5879c89f5c-6kks7\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:56:51.482891 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.482731 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-868v5\" (UniqueName: \"kubernetes.io/projected/91de2498-e273-4dea-a47b-b0914894c354-kube-api-access-868v5\") pod \"test-model-b07e06-kserve-5879c89f5c-6kks7\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:56:51.482891 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.482754 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-dshm\") pod \"test-model-b07e06-kserve-5879c89f5c-6kks7\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:56:51.482891 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.482786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-tmp-dir\") pod \"test-model-b07e06-kserve-5879c89f5c-6kks7\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:56:51.482891 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.482815 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91de2498-e273-4dea-a47b-b0914894c354-tls-certs\") pod \"test-model-b07e06-kserve-5879c89f5c-6kks7\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:56:51.483187 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.483161 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-home\") pod \"test-model-b07e06-kserve-5879c89f5c-6kks7\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:56:51.483295 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.483274 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-model-cache\") pod \"test-model-b07e06-kserve-5879c89f5c-6kks7\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:56:51.483576 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.483309 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-tmp-dir\") pod \"test-model-b07e06-kserve-5879c89f5c-6kks7\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:56:51.483703 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.483558 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-model-cache\") pod \"test-model-b07e06-kserve-5879c89f5c-6kks7\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:56:51.485264 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.485232 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-dshm\") pod \"test-model-b07e06-kserve-5879c89f5c-6kks7\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:56:51.485678 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.485660 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91de2498-e273-4dea-a47b-b0914894c354-tls-certs\") pod \"test-model-b07e06-kserve-5879c89f5c-6kks7\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:56:51.495038 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.495012 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-868v5\" (UniqueName: \"kubernetes.io/projected/91de2498-e273-4dea-a47b-b0914894c354-kube-api-access-868v5\") pod \"test-model-b07e06-kserve-5879c89f5c-6kks7\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:56:51.558892 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.558802 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:56:51.700363 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.700338 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7"] Jul 07 18:56:51.702550 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:56:51.702524 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91de2498_e273_4dea_a47b_b0914894c354.slice/crio-26ef862428c95616909c814fc0da05881044678b23a6c4b0524264840cf6ac47 WatchSource:0}: Error finding container 26ef862428c95616909c814fc0da05881044678b23a6c4b0524264840cf6ac47: Status 404 returned error can't find the container with id 26ef862428c95616909c814fc0da05881044678b23a6c4b0524264840cf6ac47 Jul 07 18:56:51.874254 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.874163 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" event={"ID":"91de2498-e273-4dea-a47b-b0914894c354","Type":"ContainerStarted","Data":"fc4a3ed895e73f08a55ec80e82496b472d5413aed9221830a8dd9518dff3d313"} Jul 07 18:56:51.874254 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:51.874208 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" event={"ID":"91de2498-e273-4dea-a47b-b0914894c354","Type":"ContainerStarted","Data":"26ef862428c95616909c814fc0da05881044678b23a6c4b0524264840cf6ac47"} Jul 07 18:56:55.259844 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:55.259789 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" podStartSLOduration=4.259768681 podStartE2EDuration="4.259768681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:56:51.924681672 +0000 UTC m=+2035.308475069" watchObservedRunningTime="2026-07-07 18:56:55.259768681 +0000 UTC m=+2038.643562077" Jul 07 18:56:55.260361 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:56:55.260345 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:57:01.559466 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:01.559428 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:57:01.559970 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:01.559501 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:57:01.573358 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:01.573333 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:57:01.931948 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:01.931867 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:57:19.210904 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:19.210818 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:57:23.311356 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:23.311307 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:57:35.770647 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:35.770608 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:57:39.728538 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:39.728508 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7"] Jul 07 18:57:39.750074 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:39.750042 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:57:40.852748 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:40.852707 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:57:41.063406 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:41.063372 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:57:41.475505 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:41.475477 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:57:41.851566 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:41.851484 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:57:41.983820 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:41.983789 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:57:42.237145 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:42.237102 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:57:46.506266 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:46.506231 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:57:52.049169 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:52.049128 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:57:54.747815 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:54.747776 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" podUID="91de2498-e273-4dea-a47b-b0914894c354" containerName="main" containerID="cri-o://fc4a3ed895e73f08a55ec80e82496b472d5413aed9221830a8dd9518dff3d313" gracePeriod=45 Jul 07 18:57:54.997056 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:54.997032 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:57:55.128029 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.127943 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91de2498-e273-4dea-a47b-b0914894c354-tls-certs\") pod \"91de2498-e273-4dea-a47b-b0914894c354\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " Jul 07 18:57:55.128029 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.127993 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-868v5\" (UniqueName: \"kubernetes.io/projected/91de2498-e273-4dea-a47b-b0914894c354-kube-api-access-868v5\") pod \"91de2498-e273-4dea-a47b-b0914894c354\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " Jul 07 18:57:55.128029 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.128015 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-home\") pod \"91de2498-e273-4dea-a47b-b0914894c354\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " Jul 07 18:57:55.128281 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.128038 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-model-cache\") pod \"91de2498-e273-4dea-a47b-b0914894c354\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " Jul 07 18:57:55.128281 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.128100 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-dshm\") pod \"91de2498-e273-4dea-a47b-b0914894c354\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " Jul 07 18:57:55.128281 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.128128 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-tmp-dir\") pod \"91de2498-e273-4dea-a47b-b0914894c354\" (UID: \"91de2498-e273-4dea-a47b-b0914894c354\") " Jul 07 18:57:55.128477 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.128285 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-home" (OuterVolumeSpecName: "home") pod "91de2498-e273-4dea-a47b-b0914894c354" (UID: "91de2498-e273-4dea-a47b-b0914894c354"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:57:55.128477 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.128356 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-model-cache" (OuterVolumeSpecName: "model-cache") pod "91de2498-e273-4dea-a47b-b0914894c354" (UID: "91de2498-e273-4dea-a47b-b0914894c354"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:57:55.128477 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.128408 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-home\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:57:55.128477 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.128461 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-model-cache\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:57:55.128654 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.128484 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "91de2498-e273-4dea-a47b-b0914894c354" (UID: "91de2498-e273-4dea-a47b-b0914894c354"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:57:55.130301 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.130267 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91de2498-e273-4dea-a47b-b0914894c354-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "91de2498-e273-4dea-a47b-b0914894c354" (UID: "91de2498-e273-4dea-a47b-b0914894c354"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:57:55.130417 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.130308 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-dshm" (OuterVolumeSpecName: "dshm") pod "91de2498-e273-4dea-a47b-b0914894c354" (UID: "91de2498-e273-4dea-a47b-b0914894c354"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:57:55.130417 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.130345 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91de2498-e273-4dea-a47b-b0914894c354-kube-api-access-868v5" (OuterVolumeSpecName: "kube-api-access-868v5") pod "91de2498-e273-4dea-a47b-b0914894c354" (UID: "91de2498-e273-4dea-a47b-b0914894c354"). InnerVolumeSpecName "kube-api-access-868v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:57:55.170434 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.170396 2578 generic.go:356] "Generic (PLEG): container finished" podID="91de2498-e273-4dea-a47b-b0914894c354" containerID="fc4a3ed895e73f08a55ec80e82496b472d5413aed9221830a8dd9518dff3d313" exitCode=0 Jul 07 18:57:55.170617 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.170446 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" Jul 07 18:57:55.170617 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.170450 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" event={"ID":"91de2498-e273-4dea-a47b-b0914894c354","Type":"ContainerDied","Data":"fc4a3ed895e73f08a55ec80e82496b472d5413aed9221830a8dd9518dff3d313"} Jul 07 18:57:55.170617 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.170493 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7" event={"ID":"91de2498-e273-4dea-a47b-b0914894c354","Type":"ContainerDied","Data":"26ef862428c95616909c814fc0da05881044678b23a6c4b0524264840cf6ac47"} Jul 07 18:57:55.170617 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.170515 2578 scope.go:117] "RemoveContainer" containerID="fc4a3ed895e73f08a55ec80e82496b472d5413aed9221830a8dd9518dff3d313" Jul 07 18:57:55.179845 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.179822 2578 scope.go:117] "RemoveContainer" containerID="fc4a3ed895e73f08a55ec80e82496b472d5413aed9221830a8dd9518dff3d313" Jul 07 18:57:55.180115 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:57:55.180097 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4a3ed895e73f08a55ec80e82496b472d5413aed9221830a8dd9518dff3d313\": container with ID starting with fc4a3ed895e73f08a55ec80e82496b472d5413aed9221830a8dd9518dff3d313 not found: ID does not exist" containerID="fc4a3ed895e73f08a55ec80e82496b472d5413aed9221830a8dd9518dff3d313" Jul 07 18:57:55.180183 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.180128 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4a3ed895e73f08a55ec80e82496b472d5413aed9221830a8dd9518dff3d313"} err="failed to get container status \"fc4a3ed895e73f08a55ec80e82496b472d5413aed9221830a8dd9518dff3d313\": rpc error: code = NotFound desc = could not find container \"fc4a3ed895e73f08a55ec80e82496b472d5413aed9221830a8dd9518dff3d313\": container with ID starting with fc4a3ed895e73f08a55ec80e82496b472d5413aed9221830a8dd9518dff3d313 not found: ID does not exist" Jul 07 18:57:55.198091 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.198068 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7"] Jul 07 18:57:55.203353 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.203330 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b07e06/test-model-b07e06-kserve-5879c89f5c-6kks7"] Jul 07 18:57:55.229730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.229709 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-dshm\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:57:55.229730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.229732 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91de2498-e273-4dea-a47b-b0914894c354-tmp-dir\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:57:55.229856 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.229741 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91de2498-e273-4dea-a47b-b0914894c354-tls-certs\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:57:55.229856 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:55.229749 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-868v5\" (UniqueName: \"kubernetes.io/projected/91de2498-e273-4dea-a47b-b0914894c354-kube-api-access-868v5\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:57:57.194370 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:57.194338 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 18:57:57.199053 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:57.199028 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91de2498-e273-4dea-a47b-b0914894c354" path="/var/lib/kubelet/pods/91de2498-e273-4dea-a47b-b0914894c354/volumes" Jul 07 18:57:57.199229 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:57.199211 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 18:57:59.782782 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:59.782738 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:57:59.891880 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:57:59.891837 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:23.972748 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:23.972715 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:24.306007 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:24.305917 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw"] Jul 07 18:58:24.306215 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:24.306156 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" podUID="f667ca82-13e6-4f98-927c-c376fec7de6d" containerName="istio-proxy" containerID="cri-o://bfc598764d718b5130e8bddf53012b71dc788fb42c7512010e39e229d12fffe0" gracePeriod=30 Jul 07 18:58:24.343468 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:24.343439 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:24.605102 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:24.605023 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:24.845728 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:24.845697 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:25.372089 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:25.372055 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:25.661022 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:25.660942 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:25.804377 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:25.804338 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:26.083790 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:26.083754 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:28.178383 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:28.178352 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:58:29.556002 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.555974 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:58:29.595967 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.595914 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-podinfo\") pod \"f667ca82-13e6-4f98-927c-c376fec7de6d\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " Jul 07 18:58:29.596075 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.595971 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f667ca82-13e6-4f98-927c-c376fec7de6d-istiod-ca-cert\") pod \"f667ca82-13e6-4f98-927c-c376fec7de6d\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " Jul 07 18:58:29.596075 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.595996 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-envoy\") pod \"f667ca82-13e6-4f98-927c-c376fec7de6d\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " Jul 07 18:58:29.596075 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.596025 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-data\") pod \"f667ca82-13e6-4f98-927c-c376fec7de6d\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " Jul 07 18:58:29.596075 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.596043 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7949\" (UniqueName: \"kubernetes.io/projected/f667ca82-13e6-4f98-927c-c376fec7de6d-kube-api-access-t7949\") pod \"f667ca82-13e6-4f98-927c-c376fec7de6d\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " Jul 07 18:58:29.596287 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.596079 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-credential-socket\") pod \"f667ca82-13e6-4f98-927c-c376fec7de6d\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " Jul 07 18:58:29.596287 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.596110 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-workload-certs\") pod \"f667ca82-13e6-4f98-927c-c376fec7de6d\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " Jul 07 18:58:29.596287 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.596138 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-token\") pod \"f667ca82-13e6-4f98-927c-c376fec7de6d\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " Jul 07 18:58:29.596287 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.596165 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-workload-socket\") pod \"f667ca82-13e6-4f98-927c-c376fec7de6d\" (UID: \"f667ca82-13e6-4f98-927c-c376fec7de6d\") " Jul 07 18:58:29.596523 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.596477 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f667ca82-13e6-4f98-927c-c376fec7de6d" (UID: "f667ca82-13e6-4f98-927c-c376fec7de6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:58:29.596579 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.596414 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f667ca82-13e6-4f98-927c-c376fec7de6d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f667ca82-13e6-4f98-927c-c376fec7de6d" (UID: "f667ca82-13e6-4f98-927c-c376fec7de6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:58:29.596579 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.596564 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f667ca82-13e6-4f98-927c-c376fec7de6d" (UID: "f667ca82-13e6-4f98-927c-c376fec7de6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:58:29.596676 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.596579 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f667ca82-13e6-4f98-927c-c376fec7de6d" (UID: "f667ca82-13e6-4f98-927c-c376fec7de6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:58:29.597123 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.597093 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-data" (OuterVolumeSpecName: "istio-data") pod "f667ca82-13e6-4f98-927c-c376fec7de6d" (UID: "f667ca82-13e6-4f98-927c-c376fec7de6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:58:29.598073 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.598043 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f667ca82-13e6-4f98-927c-c376fec7de6d" (UID: "f667ca82-13e6-4f98-927c-c376fec7de6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:58:29.598380 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.598355 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f667ca82-13e6-4f98-927c-c376fec7de6d" (UID: "f667ca82-13e6-4f98-927c-c376fec7de6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:58:29.598480 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.598435 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f667ca82-13e6-4f98-927c-c376fec7de6d-kube-api-access-t7949" (OuterVolumeSpecName: "kube-api-access-t7949") pod "f667ca82-13e6-4f98-927c-c376fec7de6d" (UID: "f667ca82-13e6-4f98-927c-c376fec7de6d"). InnerVolumeSpecName "kube-api-access-t7949". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:58:29.598533 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.598492 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-token" (OuterVolumeSpecName: "istio-token") pod "f667ca82-13e6-4f98-927c-c376fec7de6d" (UID: "f667ca82-13e6-4f98-927c-c376fec7de6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:58:29.697499 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.697477 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f667ca82-13e6-4f98-927c-c376fec7de6d-istiod-ca-cert\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:58:29.697499 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.697497 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-envoy\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:58:29.697632 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.697505 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-data\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:58:29.697632 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.697514 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7949\" (UniqueName: \"kubernetes.io/projected/f667ca82-13e6-4f98-927c-c376fec7de6d-kube-api-access-t7949\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:58:29.697632 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.697523 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-credential-socket\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:58:29.697632 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.697531 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-workload-certs\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:58:29.697632 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.697539 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-token\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:58:29.697632 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.697546 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f667ca82-13e6-4f98-927c-c376fec7de6d-workload-socket\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:58:29.697632 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:29.697554 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f667ca82-13e6-4f98-927c-c376fec7de6d-istio-podinfo\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:58:30.332599 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:30.332575 2578 generic.go:356] "Generic (PLEG): container finished" podID="f667ca82-13e6-4f98-927c-c376fec7de6d" containerID="bfc598764d718b5130e8bddf53012b71dc788fb42c7512010e39e229d12fffe0" exitCode=0 Jul 07 18:58:30.332731 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:30.332622 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" Jul 07 18:58:30.332731 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:30.332622 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" event={"ID":"f667ca82-13e6-4f98-927c-c376fec7de6d","Type":"ContainerDied","Data":"bfc598764d718b5130e8bddf53012b71dc788fb42c7512010e39e229d12fffe0"} Jul 07 18:58:30.332850 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:30.332761 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw" event={"ID":"f667ca82-13e6-4f98-927c-c376fec7de6d","Type":"ContainerDied","Data":"12014ba313c8b5e80bad6688a85da4ff7122b3c3259e5cbe5779eb104b74396a"} Jul 07 18:58:30.332850 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:30.332803 2578 scope.go:117] "RemoveContainer" containerID="bfc598764d718b5130e8bddf53012b71dc788fb42c7512010e39e229d12fffe0" Jul 07 18:58:30.341390 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:30.341373 2578 scope.go:117] "RemoveContainer" containerID="2934887cc2c4fb4f1b6d8c115aa1f67f6a218129d592abe6943aa6e45903e925" Jul 07 18:58:30.349509 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:30.349493 2578 scope.go:117] "RemoveContainer" containerID="bfc598764d718b5130e8bddf53012b71dc788fb42c7512010e39e229d12fffe0" Jul 07 18:58:30.349751 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:58:30.349733 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfc598764d718b5130e8bddf53012b71dc788fb42c7512010e39e229d12fffe0\": container with ID starting with bfc598764d718b5130e8bddf53012b71dc788fb42c7512010e39e229d12fffe0 not found: ID does not exist" containerID="bfc598764d718b5130e8bddf53012b71dc788fb42c7512010e39e229d12fffe0" Jul 07 18:58:30.349809 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:30.349758 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc598764d718b5130e8bddf53012b71dc788fb42c7512010e39e229d12fffe0"} err="failed to get container status \"bfc598764d718b5130e8bddf53012b71dc788fb42c7512010e39e229d12fffe0\": rpc error: code = NotFound desc = could not find container \"bfc598764d718b5130e8bddf53012b71dc788fb42c7512010e39e229d12fffe0\": container with ID starting with bfc598764d718b5130e8bddf53012b71dc788fb42c7512010e39e229d12fffe0 not found: ID does not exist" Jul 07 18:58:30.349809 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:30.349773 2578 scope.go:117] "RemoveContainer" containerID="2934887cc2c4fb4f1b6d8c115aa1f67f6a218129d592abe6943aa6e45903e925" Jul 07 18:58:30.349972 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:58:30.349958 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2934887cc2c4fb4f1b6d8c115aa1f67f6a218129d592abe6943aa6e45903e925\": container with ID starting with 2934887cc2c4fb4f1b6d8c115aa1f67f6a218129d592abe6943aa6e45903e925 not found: ID does not exist" containerID="2934887cc2c4fb4f1b6d8c115aa1f67f6a218129d592abe6943aa6e45903e925" Jul 07 18:58:30.350019 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:30.349974 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2934887cc2c4fb4f1b6d8c115aa1f67f6a218129d592abe6943aa6e45903e925"} err="failed to get container status \"2934887cc2c4fb4f1b6d8c115aa1f67f6a218129d592abe6943aa6e45903e925\": rpc error: code = NotFound desc = could not find container \"2934887cc2c4fb4f1b6d8c115aa1f67f6a218129d592abe6943aa6e45903e925\": container with ID starting with 2934887cc2c4fb4f1b6d8c115aa1f67f6a218129d592abe6943aa6e45903e925 not found: ID does not exist" Jul 07 18:58:30.360563 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:30.360539 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw"] Jul 07 18:58:30.365246 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:30.365226 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b07e06-openshift-default-5cf476d4f7-h4srw"] Jul 07 18:58:31.198087 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:31.198058 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f667ca82-13e6-4f98-927c-c376fec7de6d" path="/var/lib/kubelet/pods/f667ca82-13e6-4f98-927c-c376fec7de6d/volumes" Jul 07 18:58:33.045940 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:33.045909 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:33.385421 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:33.385342 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:33.631730 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:33.631686 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:33.912375 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:33.912340 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:34.124769 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.124738 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:34.520082 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.520054 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4"] Jul 07 18:58:34.520419 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.520405 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f667ca82-13e6-4f98-927c-c376fec7de6d" containerName="istio-proxy" Jul 07 18:58:34.520478 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.520421 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f667ca82-13e6-4f98-927c-c376fec7de6d" containerName="istio-proxy" Jul 07 18:58:34.520478 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.520432 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91de2498-e273-4dea-a47b-b0914894c354" containerName="main" Jul 07 18:58:34.520478 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.520437 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="91de2498-e273-4dea-a47b-b0914894c354" containerName="main" Jul 07 18:58:34.520571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.520503 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f667ca82-13e6-4f98-927c-c376fec7de6d" containerName="istio-proxy" Jul 07 18:58:34.520571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.520511 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f667ca82-13e6-4f98-927c-c376fec7de6d" containerName="istio-proxy" Jul 07 18:58:34.520571 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.520520 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="91de2498-e273-4dea-a47b-b0914894c354" containerName="main" Jul 07 18:58:34.524737 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.524709 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.530532 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.530509 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-10b7bf-openshift-default-dockercfg-n7cx5\"" Jul 07 18:58:34.572599 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.572574 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4"] Jul 07 18:58:34.635502 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.635477 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-envoy\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.635637 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.635507 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istiod-ca-cert\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.635637 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.635557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jcx4\" (UniqueName: \"kubernetes.io/projected/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-kube-api-access-4jcx4\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.635637 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.635588 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-workload-certs\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.635812 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.635655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-token\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.635812 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.635743 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-credential-socket\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.635812 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.635778 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-data\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.635938 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.635812 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-workload-socket\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.635938 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.635836 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-podinfo\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.652938 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.652899 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:34.737201 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.737173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-workload-certs\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.737344 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.737213 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-token\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.737344 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.737242 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-credential-socket\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.737344 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.737259 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-data\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.737344 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.737279 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-workload-socket\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.737344 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.737297 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-podinfo\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.737344 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.737335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-envoy\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.737660 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.737351 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istiod-ca-cert\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.737660 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.737391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jcx4\" (UniqueName: \"kubernetes.io/projected/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-kube-api-access-4jcx4\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.737769 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.737692 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-credential-socket\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.737769 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.737712 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-data\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.737769 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.737749 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-workload-socket\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.737901 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.737841 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-workload-certs\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.738207 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.738187 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istiod-ca-cert\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.739751 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.739727 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-envoy\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.739934 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.739913 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-podinfo\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.751229 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.751206 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jcx4\" (UniqueName: \"kubernetes.io/projected/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-kube-api-access-4jcx4\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.751768 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.751747 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-token\") pod \"e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.823922 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.823842 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:34.835598 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.835576 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:34.985122 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.985099 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4"] Jul 07 18:58:34.986947 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:58:34.986922 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf45af5b_0f8f_44f9_bfb2_1e4c32f3a600.slice/crio-72f54ab407bc876ecf87857e0591b4bcb8e204f84b266d1848b4f1615f9a60ec WatchSource:0}: Error finding container 72f54ab407bc876ecf87857e0591b4bcb8e204f84b266d1848b4f1615f9a60ec: Status 404 returned error can't find the container with id 72f54ab407bc876ecf87857e0591b4bcb8e204f84b266d1848b4f1615f9a60ec Jul 07 18:58:34.989046 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.989014 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:58:34.989134 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.989083 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:58:34.989134 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:34.989116 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:58:35.356584 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:35.356547 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" event={"ID":"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600","Type":"ContainerStarted","Data":"1d500e4efeb61bf8dec2ff6e0bb470f9ecb7a7ba8edc287d67e8166bb41890c2"} Jul 07 18:58:35.357049 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:35.356598 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" event={"ID":"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600","Type":"ContainerStarted","Data":"72f54ab407bc876ecf87857e0591b4bcb8e204f84b266d1848b4f1615f9a60ec"} Jul 07 18:58:35.389046 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:35.388969 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" podStartSLOduration=1.388948077 podStartE2EDuration="1.388948077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:58:35.387234272 +0000 UTC m=+2138.771027667" watchObservedRunningTime="2026-07-07 18:58:35.388948077 +0000 UTC m=+2138.772741473" Jul 07 18:58:35.836399 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:35.836361 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:36.841283 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:36.841254 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:37.367449 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:37.367419 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:37.368420 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:37.368392 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:37.599877 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:37.599846 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:37.792445 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:37.792411 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:37.973846 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:37.973814 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:38.981523 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:38.981490 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:39.403741 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:39.403666 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:42.106534 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:42.106500 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:42.613670 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:42.613639 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:43.065776 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:43.065696 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:43.345642 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:43.345566 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:44.995754 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:44.995717 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:58:46.122737 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:46.122705 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:46.353132 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:46.353102 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:46.514744 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:46.514709 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:46.967916 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:46.967831 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:47.540171 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:47.540139 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:47.703005 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:47.702977 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:50.658606 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:50.658497 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:52.983580 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:52.982872 2578 patch_prober.go:28] interesting pod/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": read tcp 10.134.0.2:43174->10.134.0.48:15021: read: connection reset by peer" start-of-body= Jul 07 18:58:52.983580 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:52.982933 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" podUID="af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": read tcp 10.134.0.2:43174->10.134.0.48:15021: read: connection reset by peer" Jul 07 18:58:53.441540 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:53.441472 2578 generic.go:356] "Generic (PLEG): container finished" podID="af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" containerID="1d500e4efeb61bf8dec2ff6e0bb470f9ecb7a7ba8edc287d67e8166bb41890c2" exitCode=0 Jul 07 18:58:53.441540 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:53.441509 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" event={"ID":"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600","Type":"ContainerDied","Data":"1d500e4efeb61bf8dec2ff6e0bb470f9ecb7a7ba8edc287d67e8166bb41890c2"} Jul 07 18:58:53.441885 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:53.441867 2578 scope.go:117] "RemoveContainer" containerID="1d500e4efeb61bf8dec2ff6e0bb470f9ecb7a7ba8edc287d67e8166bb41890c2" Jul 07 18:58:53.442878 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:53.442853 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:58:53.442941 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:53.442915 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:58:53.442985 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:53.442963 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:58:54.447856 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:54.447820 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" event={"ID":"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600","Type":"ContainerStarted","Data":"f79439cf6537f2e7befd81b2a707d36906dd21277040e19840618a9fa1367b85"} Jul 07 18:58:54.836269 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:54.836230 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:54.836552 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:54.836492 2578 patch_prober.go:28] interesting pod/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 07 18:58:54.836552 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:54.836542 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" podUID="af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 07 18:58:55.836410 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:55.836373 2578 patch_prober.go:28] interesting pod/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 07 18:58:55.836846 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:55.836431 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" podUID="af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 07 18:58:56.836187 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:56.836143 2578 patch_prober.go:28] interesting pod/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 07 18:58:56.836441 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:56.836245 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" podUID="af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 07 18:58:57.836496 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:57.836460 2578 patch_prober.go:28] interesting pod/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 07 18:58:57.836874 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:57.836512 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" podUID="af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 07 18:58:58.839782 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:58.839753 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:58.840256 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:58.840147 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:58.840803 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:58.840786 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:58:59.296220 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:59.296191 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:59.726440 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:59.726407 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:58:59.921840 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:58:59.921809 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:00.031881 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:00.031809 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:00.234829 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:00.234797 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:00.973132 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:00.973097 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:59:07.010575 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:07.010540 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:07.110896 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:07.110867 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:07.292963 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:07.292894 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4"] Jul 07 18:59:07.293110 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:07.293091 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" podUID="af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" containerName="istio-proxy" containerID="cri-o://f79439cf6537f2e7befd81b2a707d36906dd21277040e19840618a9fa1367b85" gracePeriod=30 Jul 07 18:59:07.338701 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:07.338673 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:07.585361 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:07.585279 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:07.918619 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:07.918534 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:08.287851 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:08.287812 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:08.503920 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:08.503890 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:08.596625 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:08.596559 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:08.856144 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:08.856062 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:11.445206 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:11.445176 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:59:12.533194 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.533155 2578 generic.go:356] "Generic (PLEG): container finished" podID="af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" containerID="f79439cf6537f2e7befd81b2a707d36906dd21277040e19840618a9fa1367b85" exitCode=0 Jul 07 18:59:12.533514 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.533194 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" event={"ID":"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600","Type":"ContainerDied","Data":"f79439cf6537f2e7befd81b2a707d36906dd21277040e19840618a9fa1367b85"} Jul 07 18:59:12.533514 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.533242 2578 scope.go:117] "RemoveContainer" containerID="1d500e4efeb61bf8dec2ff6e0bb470f9ecb7a7ba8edc287d67e8166bb41890c2" Jul 07 18:59:12.549635 ip-10-0-141-108 kubenswrapper[2578]: E0707 18:59:12.549615 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d500e4efeb61bf8dec2ff6e0bb470f9ecb7a7ba8edc287d67e8166bb41890c2\": container with ID starting with 1d500e4efeb61bf8dec2ff6e0bb470f9ecb7a7ba8edc287d67e8166bb41890c2 not found: ID does not exist" containerID="1d500e4efeb61bf8dec2ff6e0bb470f9ecb7a7ba8edc287d67e8166bb41890c2" Jul 07 18:59:12.553206 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.553187 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:59:12.737835 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.737811 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-token\") pod \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " Jul 07 18:59:12.737958 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.737847 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istiod-ca-cert\") pod \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " Jul 07 18:59:12.737958 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.737876 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-envoy\") pod \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " Jul 07 18:59:12.737958 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.737897 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-data\") pod \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " Jul 07 18:59:12.738141 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.738059 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-workload-socket\") pod \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " Jul 07 18:59:12.738141 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.738104 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-podinfo\") pod \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " Jul 07 18:59:12.738141 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.738135 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-workload-certs\") pod \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " Jul 07 18:59:12.738300 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.738161 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jcx4\" (UniqueName: \"kubernetes.io/projected/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-kube-api-access-4jcx4\") pod \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " Jul 07 18:59:12.738300 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.738231 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-credential-socket\") pod \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\" (UID: \"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600\") " Jul 07 18:59:12.738300 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.738244 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" (UID: "af45af5b-0f8f-44f9-bfb2-1e4c32f3a600"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:59:12.738599 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.738353 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" (UID: "af45af5b-0f8f-44f9-bfb2-1e4c32f3a600"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:59:12.738599 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.738543 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" (UID: "af45af5b-0f8f-44f9-bfb2-1e4c32f3a600"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:59:12.738753 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.738714 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" (UID: "af45af5b-0f8f-44f9-bfb2-1e4c32f3a600"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:59:12.738925 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.738901 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-data" (OuterVolumeSpecName: "istio-data") pod "af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" (UID: "af45af5b-0f8f-44f9-bfb2-1e4c32f3a600"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:59:12.740288 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.740253 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" (UID: "af45af5b-0f8f-44f9-bfb2-1e4c32f3a600"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:59:12.740436 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.740417 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-kube-api-access-4jcx4" (OuterVolumeSpecName: "kube-api-access-4jcx4") pod "af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" (UID: "af45af5b-0f8f-44f9-bfb2-1e4c32f3a600"). InnerVolumeSpecName "kube-api-access-4jcx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:59:12.740651 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.740623 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-token" (OuterVolumeSpecName: "istio-token") pod "af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" (UID: "af45af5b-0f8f-44f9-bfb2-1e4c32f3a600"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:59:12.740754 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.740654 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" (UID: "af45af5b-0f8f-44f9-bfb2-1e4c32f3a600"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:59:12.839569 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.839545 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-workload-socket\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:59:12.839569 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.839565 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-podinfo\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:59:12.839701 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.839575 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-workload-certs\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:59:12.839701 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.839583 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jcx4\" (UniqueName: \"kubernetes.io/projected/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-kube-api-access-4jcx4\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:59:12.839701 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.839593 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-credential-socket\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:59:12.839701 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.839601 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-token\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:59:12.839701 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.839609 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istiod-ca-cert\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:59:12.839701 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.839616 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-envoy\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:59:12.839701 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:12.839623 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600-istio-data\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 18:59:13.538632 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:13.538600 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" Jul 07 18:59:13.538632 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:13.538616 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4" event={"ID":"af45af5b-0f8f-44f9-bfb2-1e4c32f3a600","Type":"ContainerDied","Data":"72f54ab407bc876ecf87857e0591b4bcb8e204f84b266d1848b4f1615f9a60ec"} Jul 07 18:59:13.539123 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:13.538674 2578 scope.go:117] "RemoveContainer" containerID="f79439cf6537f2e7befd81b2a707d36906dd21277040e19840618a9fa1367b85" Jul 07 18:59:13.560463 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:13.560434 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4"] Jul 07 18:59:13.567899 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:13.567877 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-10b7bf-openshift-default-7669b88547-bcqm4"] Jul 07 18:59:14.548466 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:14.548439 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:14.650887 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:14.650856 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:14.832479 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:14.832402 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:15.074753 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.074724 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:15.197728 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.197647 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" path="/var/lib/kubelet/pods/af45af5b-0f8f-44f9-bfb2-1e4c32f3a600/volumes" Jul 07 18:59:15.496333 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.496287 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:15.867516 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.867438 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2"] Jul 07 18:59:15.867856 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.867775 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" containerName="istio-proxy" Jul 07 18:59:15.867856 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.867787 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" containerName="istio-proxy" Jul 07 18:59:15.867856 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.867807 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f667ca82-13e6-4f98-927c-c376fec7de6d" containerName="istio-proxy" Jul 07 18:59:15.867856 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.867813 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f667ca82-13e6-4f98-927c-c376fec7de6d" containerName="istio-proxy" Jul 07 18:59:15.867985 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.867871 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" containerName="istio-proxy" Jul 07 18:59:15.867985 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.867880 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" containerName="istio-proxy" Jul 07 18:59:15.872142 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.872116 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:15.874957 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.874937 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-972159-openshift-default-dockercfg-9qt2d\"" Jul 07 18:59:15.896088 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.896066 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2"] Jul 07 18:59:15.919184 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.919158 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:15.961887 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.961863 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwhbg\" (UniqueName: \"kubernetes.io/projected/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-kube-api-access-gwhbg\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:15.962081 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.961895 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-envoy\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:15.962081 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.961921 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-workload-socket\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:15.962081 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.961936 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-workload-certs\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:15.962081 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.961952 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-podinfo\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:15.962081 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.961973 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-token\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:15.962081 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.961994 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istiod-ca-cert\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:15.962489 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.962100 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-data\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:15.962489 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:15.962147 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-credential-socket\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:16.062670 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.062643 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-credential-socket\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:16.062802 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.062684 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwhbg\" (UniqueName: \"kubernetes.io/projected/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-kube-api-access-gwhbg\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:16.062802 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.062734 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-envoy\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:16.062802 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.062772 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-workload-socket\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:16.062916 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.062896 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-workload-certs\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:16.062961 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.062931 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-podinfo\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:16.063002 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.062972 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-token\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:16.063050 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.063009 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istiod-ca-cert\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:16.063106 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.063076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-data\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:16.063106 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.063097 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-workload-socket\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:16.063205 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.063074 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-credential-socket\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:16.063383 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.063362 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-data\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:16.063499 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.063472 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-workload-certs\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:16.063775 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.063755 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istiod-ca-cert\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:16.065083 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.065064 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-envoy\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:16.065394 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.065366 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-podinfo\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:16.075934 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.075910 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwhbg\" (UniqueName: \"kubernetes.io/projected/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-kube-api-access-gwhbg\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:16.076413 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.076395 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-token\") pod \"e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:16.182617 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.182533 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:16.340657 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.340631 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2"] Jul 07 18:59:16.342695 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:59:16.342657 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5b7e47_9f2b_4200_aa8f_8cbf06ba08c2.slice/crio-6c6daed4d27d4f95e4ea3587b88c923898ad9af2971f39b92ecd286365223e8e WatchSource:0}: Error finding container 6c6daed4d27d4f95e4ea3587b88c923898ad9af2971f39b92ecd286365223e8e: Status 404 returned error can't find the container with id 6c6daed4d27d4f95e4ea3587b88c923898ad9af2971f39b92ecd286365223e8e Jul 07 18:59:16.344013 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.343971 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:16.344992 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.344962 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:59:16.345081 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.345027 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:59:16.345081 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.345056 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:59:16.553547 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.553510 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" event={"ID":"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2","Type":"ContainerStarted","Data":"288f1a74bf1074f995e10d82428d9c9c7c82a1f14e73e7e5645e2a59ee4785d7"} Jul 07 18:59:16.553737 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.553563 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" event={"ID":"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2","Type":"ContainerStarted","Data":"6c6daed4d27d4f95e4ea3587b88c923898ad9af2971f39b92ecd286365223e8e"} Jul 07 18:59:16.588044 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:16.587997 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" podStartSLOduration=1.587982331 podStartE2EDuration="1.587982331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:59:16.585272168 +0000 UTC m=+2179.969065563" watchObservedRunningTime="2026-07-07 18:59:16.587982331 +0000 UTC m=+2179.971775725" Jul 07 18:59:17.183360 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:17.183307 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:17.188119 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:17.188096 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:17.558409 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:17.558378 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:17.559427 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:17.559404 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:19.045263 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:19.045232 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:19.352793 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:19.352706 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:19.535732 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:19.535704 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:19.717494 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:19.717386 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:20.258816 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:20.258785 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:20.470783 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:20.470733 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:23.336309 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:23.336263 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:23.708236 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:23.708154 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:24.104725 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:24.104665 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:24.277458 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:24.277423 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:24.376225 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:24.376139 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:59:27.209003 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:27.208557 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:27.354021 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:27.353970 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:27.451061 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:27.451013 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:27.990155 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:27.990120 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:28.091704 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:28.091672 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:31.757522 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:31.757484 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:32.559597 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:32.559570 2578 patch_prober.go:28] interesting pod/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 07 18:59:32.559788 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:32.559618 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" podUID="cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 07 18:59:32.628748 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:32.628718 2578 generic.go:356] "Generic (PLEG): container finished" podID="cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" containerID="288f1a74bf1074f995e10d82428d9c9c7c82a1f14e73e7e5645e2a59ee4785d7" exitCode=0 Jul 07 18:59:32.628905 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:32.628796 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" event={"ID":"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2","Type":"ContainerDied","Data":"288f1a74bf1074f995e10d82428d9c9c7c82a1f14e73e7e5645e2a59ee4785d7"} Jul 07 18:59:32.629156 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:32.629140 2578 scope.go:117] "RemoveContainer" containerID="288f1a74bf1074f995e10d82428d9c9c7c82a1f14e73e7e5645e2a59ee4785d7" Jul 07 18:59:32.630176 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:32.630151 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:59:32.630247 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:32.630230 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:59:32.630295 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:32.630280 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 18:59:33.635199 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:33.635158 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" event={"ID":"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2","Type":"ContainerStarted","Data":"67629322e27949f773cdb750014f8e882ba2b8a821837261c183e6235f2ee0cf"} Jul 07 18:59:34.183839 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:34.183805 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:34.184240 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:34.184141 2578 patch_prober.go:28] interesting pod/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 07 18:59:34.184451 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:34.184427 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" podUID="cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 07 18:59:35.183971 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:35.183929 2578 patch_prober.go:28] interesting pod/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 07 18:59:35.184486 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:35.183994 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" podUID="cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 07 18:59:35.964510 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:35.964480 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf"] Jul 07 18:59:35.964950 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:35.964927 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" containerName="istio-proxy" Jul 07 18:59:35.964950 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:35.964951 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="af45af5b-0f8f-44f9-bfb2-1e4c32f3a600" containerName="istio-proxy" Jul 07 18:59:35.966960 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:35.966940 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:35.971597 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:35.971572 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-972159\"/\"kube-root-ca.crt\"" Jul 07 18:59:35.975758 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:35.975729 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-972159\"/\"auth-test-model-972159-kserve-self-signed-certs\"" Jul 07 18:59:35.976264 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:35.976240 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-972159\"/\"default-dockercfg-rpkzl\"" Jul 07 18:59:35.976644 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:35.976631 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-972159\"/\"openshift-service-ca.crt\"" Jul 07 18:59:35.982778 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:35.982751 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf"] Jul 07 18:59:36.006255 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.006228 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:36.011156 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.011133 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-dshm\") pod \"auth-test-model-972159-kserve-65dd9b4887-sk8kf\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:36.011289 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.011161 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-tmp-dir\") pod \"auth-test-model-972159-kserve-65dd9b4887-sk8kf\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:36.011289 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.011191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkd6m\" (UniqueName: \"kubernetes.io/projected/4a13007e-8590-4d14-b1a1-b55455a1b9f5-kube-api-access-mkd6m\") pod \"auth-test-model-972159-kserve-65dd9b4887-sk8kf\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:36.011289 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.011244 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-home\") pod \"auth-test-model-972159-kserve-65dd9b4887-sk8kf\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:36.011497 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.011358 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a13007e-8590-4d14-b1a1-b55455a1b9f5-tls-certs\") pod \"auth-test-model-972159-kserve-65dd9b4887-sk8kf\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:36.011497 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.011388 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-model-cache\") pod \"auth-test-model-972159-kserve-65dd9b4887-sk8kf\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:36.112292 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.112252 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-model-cache\") pod \"auth-test-model-972159-kserve-65dd9b4887-sk8kf\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:36.112503 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.112307 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-dshm\") pod \"auth-test-model-972159-kserve-65dd9b4887-sk8kf\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:36.112503 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.112359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-tmp-dir\") pod \"auth-test-model-972159-kserve-65dd9b4887-sk8kf\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:36.112503 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.112401 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkd6m\" (UniqueName: \"kubernetes.io/projected/4a13007e-8590-4d14-b1a1-b55455a1b9f5-kube-api-access-mkd6m\") pod \"auth-test-model-972159-kserve-65dd9b4887-sk8kf\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:36.112503 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.112430 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-home\") pod \"auth-test-model-972159-kserve-65dd9b4887-sk8kf\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:36.112699 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.112533 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a13007e-8590-4d14-b1a1-b55455a1b9f5-tls-certs\") pod \"auth-test-model-972159-kserve-65dd9b4887-sk8kf\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:36.112747 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.112709 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-model-cache\") pod \"auth-test-model-972159-kserve-65dd9b4887-sk8kf\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:36.112821 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.112801 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-tmp-dir\") pod \"auth-test-model-972159-kserve-65dd9b4887-sk8kf\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:36.112858 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.112828 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-home\") pod \"auth-test-model-972159-kserve-65dd9b4887-sk8kf\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:36.114874 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.114850 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-dshm\") pod \"auth-test-model-972159-kserve-65dd9b4887-sk8kf\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:36.115086 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.115069 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a13007e-8590-4d14-b1a1-b55455a1b9f5-tls-certs\") pod \"auth-test-model-972159-kserve-65dd9b4887-sk8kf\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:36.121887 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.121865 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkd6m\" (UniqueName: \"kubernetes.io/projected/4a13007e-8590-4d14-b1a1-b55455a1b9f5-kube-api-access-mkd6m\") pod \"auth-test-model-972159-kserve-65dd9b4887-sk8kf\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:36.183870 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.183831 2578 patch_prober.go:28] interesting pod/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 07 18:59:36.184043 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.183895 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" podUID="cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 07 18:59:36.293004 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.292973 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:36.460591 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.460567 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf"] Jul 07 18:59:36.462090 ip-10-0-141-108 kubenswrapper[2578]: W0707 18:59:36.462064 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a13007e_8590_4d14_b1a1_b55455a1b9f5.slice/crio-de808ac04c20d6d438f8efb67a853a49846fa53fb570e49e89fdbcde11ba8bc1 WatchSource:0}: Error finding container de808ac04c20d6d438f8efb67a853a49846fa53fb570e49e89fdbcde11ba8bc1: Status 404 returned error can't find the container with id de808ac04c20d6d438f8efb67a853a49846fa53fb570e49e89fdbcde11ba8bc1 Jul 07 18:59:36.650367 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.650265 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" event={"ID":"4a13007e-8590-4d14-b1a1-b55455a1b9f5","Type":"ContainerStarted","Data":"915b345cc09ec63a56d91c8d0a5fc74b54e4516b14d29fb2388e0a8ed200c3ae"} Jul 07 18:59:36.650367 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.650303 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" event={"ID":"4a13007e-8590-4d14-b1a1-b55455a1b9f5","Type":"ContainerStarted","Data":"de808ac04c20d6d438f8efb67a853a49846fa53fb570e49e89fdbcde11ba8bc1"} Jul 07 18:59:36.695999 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:36.695869 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:37.183807 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:37.183766 2578 patch_prober.go:28] interesting pod/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 07 18:59:37.184001 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:37.183829 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" podUID="cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 07 18:59:38.186596 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:38.186565 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:38.186992 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:38.186938 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:38.187579 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:38.187559 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 18:59:38.213873 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:38.213834 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" podStartSLOduration=3.213822346 podStartE2EDuration="3.213822346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:59:36.676351215 +0000 UTC m=+2200.060144610" watchObservedRunningTime="2026-07-07 18:59:38.213822346 +0000 UTC m=+2201.597615741" Jul 07 18:59:39.453734 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:39.453703 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 18:59:41.402185 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:41.402150 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:41.506777 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:41.506747 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:46.294135 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:46.294103 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:46.294641 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:46.294157 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:46.306233 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:46.306205 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:46.713640 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:46.713561 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 18:59:50.084483 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:50.084446 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 18:59:50.209510 ip-10-0-141-108 kubenswrapper[2578]: I0707 18:59:50.209477 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:00:02.050224 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.050192 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt"] Jul 07 19:00:02.053469 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.053454 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:02.057137 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.057113 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e7285b\"/\"auth-test-model-e7285b-kserve-self-signed-certs\"" Jul 07 19:00:02.058089 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.058072 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e7285b\"/\"kube-root-ca.crt\"" Jul 07 19:00:02.058181 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.058139 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e7285b\"/\"openshift-service-ca.crt\"" Jul 07 19:00:02.058242 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.058217 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e7285b\"/\"default-dockercfg-5t9b6\"" Jul 07 19:00:02.068609 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.068587 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt"] Jul 07 19:00:02.116892 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.116854 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:00:02.119570 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.119553 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-home\") pod \"auth-test-model-e7285b-kserve-768f8dc567-hqblt\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:02.119644 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.119582 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbn4f\" (UniqueName: \"kubernetes.io/projected/c3d99b95-5ee8-486d-bd52-b3270cf72980-kube-api-access-pbn4f\") pod \"auth-test-model-e7285b-kserve-768f8dc567-hqblt\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:02.119644 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.119632 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3d99b95-5ee8-486d-bd52-b3270cf72980-tls-certs\") pod \"auth-test-model-e7285b-kserve-768f8dc567-hqblt\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:02.119761 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.119653 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-model-cache\") pod \"auth-test-model-e7285b-kserve-768f8dc567-hqblt\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:02.119761 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.119674 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-tmp-dir\") pod \"auth-test-model-e7285b-kserve-768f8dc567-hqblt\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:02.119761 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.119699 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-dshm\") pod \"auth-test-model-e7285b-kserve-768f8dc567-hqblt\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:02.220507 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.220484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3d99b95-5ee8-486d-bd52-b3270cf72980-tls-certs\") pod \"auth-test-model-e7285b-kserve-768f8dc567-hqblt\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:02.220647 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.220517 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-model-cache\") pod \"auth-test-model-e7285b-kserve-768f8dc567-hqblt\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:02.220647 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.220537 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-tmp-dir\") pod \"auth-test-model-e7285b-kserve-768f8dc567-hqblt\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:02.220647 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.220568 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-dshm\") pod \"auth-test-model-e7285b-kserve-768f8dc567-hqblt\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:02.220647 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.220614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-home\") pod \"auth-test-model-e7285b-kserve-768f8dc567-hqblt\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:02.220647 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.220644 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbn4f\" (UniqueName: \"kubernetes.io/projected/c3d99b95-5ee8-486d-bd52-b3270cf72980-kube-api-access-pbn4f\") pod \"auth-test-model-e7285b-kserve-768f8dc567-hqblt\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:02.220959 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.220939 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-model-cache\") pod \"auth-test-model-e7285b-kserve-768f8dc567-hqblt\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:02.221011 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.220984 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-home\") pod \"auth-test-model-e7285b-kserve-768f8dc567-hqblt\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:02.221050 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.221010 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-tmp-dir\") pod \"auth-test-model-e7285b-kserve-768f8dc567-hqblt\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:02.222768 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.222745 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-dshm\") pod \"auth-test-model-e7285b-kserve-768f8dc567-hqblt\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:02.223042 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.223027 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3d99b95-5ee8-486d-bd52-b3270cf72980-tls-certs\") pod \"auth-test-model-e7285b-kserve-768f8dc567-hqblt\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:02.231374 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.231345 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbn4f\" (UniqueName: \"kubernetes.io/projected/c3d99b95-5ee8-486d-bd52-b3270cf72980-kube-api-access-pbn4f\") pod \"auth-test-model-e7285b-kserve-768f8dc567-hqblt\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:02.362509 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.362442 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:02.702950 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.702928 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt"] Jul 07 19:00:02.705777 ip-10-0-141-108 kubenswrapper[2578]: W0707 19:00:02.705747 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3d99b95_5ee8_486d_bd52_b3270cf72980.slice/crio-5163eeebcfa28cded23d021aa0b081f234e0c00b9da12e5d4b90245de12ac95d WatchSource:0}: Error finding container 5163eeebcfa28cded23d021aa0b081f234e0c00b9da12e5d4b90245de12ac95d: Status 404 returned error can't find the container with id 5163eeebcfa28cded23d021aa0b081f234e0c00b9da12e5d4b90245de12ac95d Jul 07 19:00:02.777366 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.777316 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" event={"ID":"c3d99b95-5ee8-486d-bd52-b3270cf72980","Type":"ContainerStarted","Data":"36594ea109cf96e441105ef7973005d62745b7be4fd96530e2e08cef6ae818ed"} Jul 07 19:00:02.777477 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.777384 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" event={"ID":"c3d99b95-5ee8-486d-bd52-b3270cf72980","Type":"ContainerStarted","Data":"5163eeebcfa28cded23d021aa0b081f234e0c00b9da12e5d4b90245de12ac95d"} Jul 07 19:00:02.802092 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:02.802044 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" podStartSLOduration=0.802021486 podStartE2EDuration="802.021486ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:02.798180089 +0000 UTC m=+2226.181973484" watchObservedRunningTime="2026-07-07 19:00:02.802021486 +0000 UTC m=+2226.185814880" Jul 07 19:00:05.844012 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:05.843972 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 19:00:12.363397 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:12.363354 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:12.363397 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:12.363406 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:12.375944 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:12.375921 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:12.832556 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:12.832530 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:00:18.854932 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:18.854837 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 19:00:26.843019 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:26.842971 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:00:36.181403 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:36.181369 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 19:00:51.676394 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:00:51.676358 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 19:01:07.262966 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:01:07.262936 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 19:01:23.949110 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:01:23.949076 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 19:01:37.142055 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:01:37.142016 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 19:01:48.973821 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:01:48.973737 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf"] Jul 07 19:01:49.017180 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:01:49.017152 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:49.923445 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:01:49.923399 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt"] Jul 07 19:01:50.355896 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:01:50.355862 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:53.343404 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:01:53.343373 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 19:02:03.993706 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:03.993648 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" podUID="4a13007e-8590-4d14-b1a1-b55455a1b9f5" containerName="main" containerID="cri-o://915b345cc09ec63a56d91c8d0a5fc74b54e4516b14d29fb2388e0a8ed200c3ae" gracePeriod=45 Jul 07 19:02:04.260284 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.260261 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 19:02:04.327293 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.327249 2578 generic.go:356] "Generic (PLEG): container finished" podID="4a13007e-8590-4d14-b1a1-b55455a1b9f5" containerID="915b345cc09ec63a56d91c8d0a5fc74b54e4516b14d29fb2388e0a8ed200c3ae" exitCode=0 Jul 07 19:02:04.327463 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.327301 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" Jul 07 19:02:04.327463 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.327333 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" event={"ID":"4a13007e-8590-4d14-b1a1-b55455a1b9f5","Type":"ContainerDied","Data":"915b345cc09ec63a56d91c8d0a5fc74b54e4516b14d29fb2388e0a8ed200c3ae"} Jul 07 19:02:04.327463 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.327364 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf" event={"ID":"4a13007e-8590-4d14-b1a1-b55455a1b9f5","Type":"ContainerDied","Data":"de808ac04c20d6d438f8efb67a853a49846fa53fb570e49e89fdbcde11ba8bc1"} Jul 07 19:02:04.327463 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.327380 2578 scope.go:117] "RemoveContainer" containerID="915b345cc09ec63a56d91c8d0a5fc74b54e4516b14d29fb2388e0a8ed200c3ae" Jul 07 19:02:04.335268 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.335248 2578 scope.go:117] "RemoveContainer" containerID="915b345cc09ec63a56d91c8d0a5fc74b54e4516b14d29fb2388e0a8ed200c3ae" Jul 07 19:02:04.335457 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.335433 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a13007e-8590-4d14-b1a1-b55455a1b9f5-tls-certs\") pod \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " Jul 07 19:02:04.335553 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.335472 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-home\") pod \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " Jul 07 19:02:04.335553 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.335500 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-model-cache\") pod \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " Jul 07 19:02:04.335553 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.335524 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-dshm\") pod \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " Jul 07 19:02:04.335695 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.335570 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkd6m\" (UniqueName: \"kubernetes.io/projected/4a13007e-8590-4d14-b1a1-b55455a1b9f5-kube-api-access-mkd6m\") pod \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " Jul 07 19:02:04.335695 ip-10-0-141-108 kubenswrapper[2578]: E0707 19:02:04.335627 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"915b345cc09ec63a56d91c8d0a5fc74b54e4516b14d29fb2388e0a8ed200c3ae\": container with ID starting with 915b345cc09ec63a56d91c8d0a5fc74b54e4516b14d29fb2388e0a8ed200c3ae not found: ID does not exist" containerID="915b345cc09ec63a56d91c8d0a5fc74b54e4516b14d29fb2388e0a8ed200c3ae" Jul 07 19:02:04.335695 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.335641 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-tmp-dir\") pod \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\" (UID: \"4a13007e-8590-4d14-b1a1-b55455a1b9f5\") " Jul 07 19:02:04.336004 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.335659 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"915b345cc09ec63a56d91c8d0a5fc74b54e4516b14d29fb2388e0a8ed200c3ae"} err="failed to get container status \"915b345cc09ec63a56d91c8d0a5fc74b54e4516b14d29fb2388e0a8ed200c3ae\": rpc error: code = NotFound desc = could not find container \"915b345cc09ec63a56d91c8d0a5fc74b54e4516b14d29fb2388e0a8ed200c3ae\": container with ID starting with 915b345cc09ec63a56d91c8d0a5fc74b54e4516b14d29fb2388e0a8ed200c3ae not found: ID does not exist" Jul 07 19:02:04.336119 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.335783 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-home" (OuterVolumeSpecName: "home") pod "4a13007e-8590-4d14-b1a1-b55455a1b9f5" (UID: "4a13007e-8590-4d14-b1a1-b55455a1b9f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:02:04.336119 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.335815 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a13007e-8590-4d14-b1a1-b55455a1b9f5" (UID: "4a13007e-8590-4d14-b1a1-b55455a1b9f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:02:04.336119 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.336016 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a13007e-8590-4d14-b1a1-b55455a1b9f5" (UID: "4a13007e-8590-4d14-b1a1-b55455a1b9f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:02:04.337575 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.337547 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a13007e-8590-4d14-b1a1-b55455a1b9f5-kube-api-access-mkd6m" (OuterVolumeSpecName: "kube-api-access-mkd6m") pod "4a13007e-8590-4d14-b1a1-b55455a1b9f5" (UID: "4a13007e-8590-4d14-b1a1-b55455a1b9f5"). InnerVolumeSpecName "kube-api-access-mkd6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:02:04.337644 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.337577 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a13007e-8590-4d14-b1a1-b55455a1b9f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a13007e-8590-4d14-b1a1-b55455a1b9f5" (UID: "4a13007e-8590-4d14-b1a1-b55455a1b9f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:02:04.337644 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.337617 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-dshm" (OuterVolumeSpecName: "dshm") pod "4a13007e-8590-4d14-b1a1-b55455a1b9f5" (UID: "4a13007e-8590-4d14-b1a1-b55455a1b9f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:02:04.436301 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.436278 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-tmp-dir\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:02:04.436301 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.436299 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a13007e-8590-4d14-b1a1-b55455a1b9f5-tls-certs\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:02:04.436436 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.436307 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-home\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:02:04.436436 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.436315 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-model-cache\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:02:04.436436 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.436340 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a13007e-8590-4d14-b1a1-b55455a1b9f5-dshm\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:02:04.436436 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.436347 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkd6m\" (UniqueName: \"kubernetes.io/projected/4a13007e-8590-4d14-b1a1-b55455a1b9f5-kube-api-access-mkd6m\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:02:04.648454 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.648431 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf"] Jul 07 19:02:04.654004 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.653985 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-972159/auth-test-model-972159-kserve-65dd9b4887-sk8kf"] Jul 07 19:02:04.943839 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:04.943754 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" podUID="c3d99b95-5ee8-486d-bd52-b3270cf72980" containerName="main" containerID="cri-o://36594ea109cf96e441105ef7973005d62745b7be4fd96530e2e08cef6ae818ed" gracePeriod=45 Jul 07 19:02:05.190288 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.190266 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:02:05.197534 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.197483 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a13007e-8590-4d14-b1a1-b55455a1b9f5" path="/var/lib/kubelet/pods/4a13007e-8590-4d14-b1a1-b55455a1b9f5/volumes" Jul 07 19:02:05.243297 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.243275 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-home\") pod \"c3d99b95-5ee8-486d-bd52-b3270cf72980\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " Jul 07 19:02:05.243432 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.243310 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3d99b95-5ee8-486d-bd52-b3270cf72980-tls-certs\") pod \"c3d99b95-5ee8-486d-bd52-b3270cf72980\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " Jul 07 19:02:05.243432 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.243357 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-dshm\") pod \"c3d99b95-5ee8-486d-bd52-b3270cf72980\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " Jul 07 19:02:05.243432 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.243394 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-model-cache\") pod \"c3d99b95-5ee8-486d-bd52-b3270cf72980\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " Jul 07 19:02:05.243432 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.243423 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-tmp-dir\") pod \"c3d99b95-5ee8-486d-bd52-b3270cf72980\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " Jul 07 19:02:05.243640 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.243467 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbn4f\" (UniqueName: \"kubernetes.io/projected/c3d99b95-5ee8-486d-bd52-b3270cf72980-kube-api-access-pbn4f\") pod \"c3d99b95-5ee8-486d-bd52-b3270cf72980\" (UID: \"c3d99b95-5ee8-486d-bd52-b3270cf72980\") " Jul 07 19:02:05.243640 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.243526 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-home" (OuterVolumeSpecName: "home") pod "c3d99b95-5ee8-486d-bd52-b3270cf72980" (UID: "c3d99b95-5ee8-486d-bd52-b3270cf72980"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:02:05.243743 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.243642 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3d99b95-5ee8-486d-bd52-b3270cf72980" (UID: "c3d99b95-5ee8-486d-bd52-b3270cf72980"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:02:05.243743 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.243689 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-home\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:02:05.243743 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.243714 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3d99b95-5ee8-486d-bd52-b3270cf72980" (UID: "c3d99b95-5ee8-486d-bd52-b3270cf72980"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:02:05.245476 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.245449 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3d99b95-5ee8-486d-bd52-b3270cf72980-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3d99b95-5ee8-486d-bd52-b3270cf72980" (UID: "c3d99b95-5ee8-486d-bd52-b3270cf72980"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:02:05.245565 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.245492 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d99b95-5ee8-486d-bd52-b3270cf72980-kube-api-access-pbn4f" (OuterVolumeSpecName: "kube-api-access-pbn4f") pod "c3d99b95-5ee8-486d-bd52-b3270cf72980" (UID: "c3d99b95-5ee8-486d-bd52-b3270cf72980"). InnerVolumeSpecName "kube-api-access-pbn4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:02:05.245565 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.245520 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-dshm" (OuterVolumeSpecName: "dshm") pod "c3d99b95-5ee8-486d-bd52-b3270cf72980" (UID: "c3d99b95-5ee8-486d-bd52-b3270cf72980"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:02:05.332521 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.332493 2578 generic.go:356] "Generic (PLEG): container finished" podID="c3d99b95-5ee8-486d-bd52-b3270cf72980" containerID="36594ea109cf96e441105ef7973005d62745b7be4fd96530e2e08cef6ae818ed" exitCode=0 Jul 07 19:02:05.332648 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.332533 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" Jul 07 19:02:05.332648 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.332578 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" event={"ID":"c3d99b95-5ee8-486d-bd52-b3270cf72980","Type":"ContainerDied","Data":"36594ea109cf96e441105ef7973005d62745b7be4fd96530e2e08cef6ae818ed"} Jul 07 19:02:05.332648 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.332614 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt" event={"ID":"c3d99b95-5ee8-486d-bd52-b3270cf72980","Type":"ContainerDied","Data":"5163eeebcfa28cded23d021aa0b081f234e0c00b9da12e5d4b90245de12ac95d"} Jul 07 19:02:05.332648 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.332638 2578 scope.go:117] "RemoveContainer" containerID="36594ea109cf96e441105ef7973005d62745b7be4fd96530e2e08cef6ae818ed" Jul 07 19:02:05.340995 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.340975 2578 scope.go:117] "RemoveContainer" containerID="36594ea109cf96e441105ef7973005d62745b7be4fd96530e2e08cef6ae818ed" Jul 07 19:02:05.341241 ip-10-0-141-108 kubenswrapper[2578]: E0707 19:02:05.341224 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36594ea109cf96e441105ef7973005d62745b7be4fd96530e2e08cef6ae818ed\": container with ID starting with 36594ea109cf96e441105ef7973005d62745b7be4fd96530e2e08cef6ae818ed not found: ID does not exist" containerID="36594ea109cf96e441105ef7973005d62745b7be4fd96530e2e08cef6ae818ed" Jul 07 19:02:05.341287 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.341249 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36594ea109cf96e441105ef7973005d62745b7be4fd96530e2e08cef6ae818ed"} err="failed to get container status \"36594ea109cf96e441105ef7973005d62745b7be4fd96530e2e08cef6ae818ed\": rpc error: code = NotFound desc = could not find container \"36594ea109cf96e441105ef7973005d62745b7be4fd96530e2e08cef6ae818ed\": container with ID starting with 36594ea109cf96e441105ef7973005d62745b7be4fd96530e2e08cef6ae818ed not found: ID does not exist" Jul 07 19:02:05.344181 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.344167 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbn4f\" (UniqueName: \"kubernetes.io/projected/c3d99b95-5ee8-486d-bd52-b3270cf72980-kube-api-access-pbn4f\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:02:05.344226 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.344184 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3d99b95-5ee8-486d-bd52-b3270cf72980-tls-certs\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:02:05.344226 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.344194 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-dshm\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:02:05.344226 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.344201 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-model-cache\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:02:05.344226 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.344209 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d99b95-5ee8-486d-bd52-b3270cf72980-tmp-dir\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:02:05.357166 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.357143 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt"] Jul 07 19:02:05.360492 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.360470 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e7285b/auth-test-model-e7285b-kserve-768f8dc567-hqblt"] Jul 07 19:02:05.749033 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.749007 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm"] Jul 07 19:02:05.749381 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.749367 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a13007e-8590-4d14-b1a1-b55455a1b9f5" containerName="main" Jul 07 19:02:05.749437 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.749382 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a13007e-8590-4d14-b1a1-b55455a1b9f5" containerName="main" Jul 07 19:02:05.749437 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.749393 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3d99b95-5ee8-486d-bd52-b3270cf72980" containerName="main" Jul 07 19:02:05.749437 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.749399 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3d99b95-5ee8-486d-bd52-b3270cf72980" containerName="main" Jul 07 19:02:05.749530 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.749458 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3d99b95-5ee8-486d-bd52-b3270cf72980" containerName="main" Jul 07 19:02:05.749530 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.749465 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a13007e-8590-4d14-b1a1-b55455a1b9f5" containerName="main" Jul 07 19:02:05.753740 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.753723 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:05.756200 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.756179 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-972159\"/\"kube-root-ca.crt\"" Jul 07 19:02:05.756436 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.756413 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-972159\"/\"openshift-service-ca.crt\"" Jul 07 19:02:05.756504 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.756440 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-972159\"/\"default-dockercfg-rpkzl\"" Jul 07 19:02:05.757308 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.757292 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-972159\"/\"sub-test-model-972159-kserve-self-signed-certs\"" Jul 07 19:02:05.769186 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.769162 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm"] Jul 07 19:02:05.790655 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.790632 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:05.849230 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.849209 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tvqh\" (UniqueName: \"kubernetes.io/projected/5b053333-9f76-412d-98c1-21a6e3a1d8e1-kube-api-access-4tvqh\") pod \"sub-test-model-972159-kserve-774cc75545-hjtgm\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:05.849365 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.849244 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-tmp-dir\") pod \"sub-test-model-972159-kserve-774cc75545-hjtgm\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:05.849365 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.849279 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-dshm\") pod \"sub-test-model-972159-kserve-774cc75545-hjtgm\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:05.849365 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.849311 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-home\") pod \"sub-test-model-972159-kserve-774cc75545-hjtgm\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:05.849498 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.849395 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b053333-9f76-412d-98c1-21a6e3a1d8e1-tls-certs\") pod \"sub-test-model-972159-kserve-774cc75545-hjtgm\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:05.849498 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.849434 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-model-cache\") pod \"sub-test-model-972159-kserve-774cc75545-hjtgm\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:05.950532 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.950501 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tvqh\" (UniqueName: \"kubernetes.io/projected/5b053333-9f76-412d-98c1-21a6e3a1d8e1-kube-api-access-4tvqh\") pod \"sub-test-model-972159-kserve-774cc75545-hjtgm\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:05.950658 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.950544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-tmp-dir\") pod \"sub-test-model-972159-kserve-774cc75545-hjtgm\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:05.950658 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.950577 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-dshm\") pod \"sub-test-model-972159-kserve-774cc75545-hjtgm\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:05.950658 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.950610 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-home\") pod \"sub-test-model-972159-kserve-774cc75545-hjtgm\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:05.950658 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.950644 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b053333-9f76-412d-98c1-21a6e3a1d8e1-tls-certs\") pod \"sub-test-model-972159-kserve-774cc75545-hjtgm\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:05.950863 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.950677 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-model-cache\") pod \"sub-test-model-972159-kserve-774cc75545-hjtgm\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:05.950970 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.950944 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-tmp-dir\") pod \"sub-test-model-972159-kserve-774cc75545-hjtgm\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:05.951078 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.951060 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-home\") pod \"sub-test-model-972159-kserve-774cc75545-hjtgm\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:05.951123 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.951102 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-model-cache\") pod \"sub-test-model-972159-kserve-774cc75545-hjtgm\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:05.952788 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.952767 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-dshm\") pod \"sub-test-model-972159-kserve-774cc75545-hjtgm\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:05.952935 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.952917 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b053333-9f76-412d-98c1-21a6e3a1d8e1-tls-certs\") pod \"sub-test-model-972159-kserve-774cc75545-hjtgm\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:05.959734 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:05.959716 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tvqh\" (UniqueName: \"kubernetes.io/projected/5b053333-9f76-412d-98c1-21a6e3a1d8e1-kube-api-access-4tvqh\") pod \"sub-test-model-972159-kserve-774cc75545-hjtgm\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:06.063042 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:06.062969 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:06.197205 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:06.197183 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm"] Jul 07 19:02:06.199231 ip-10-0-141-108 kubenswrapper[2578]: W0707 19:02:06.199207 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b053333_9f76_412d_98c1_21a6e3a1d8e1.slice/crio-c426398600eb16c9b14d7884630c38f76b90ec46f708e1a2e1a47f13648ab9c0 WatchSource:0}: Error finding container c426398600eb16c9b14d7884630c38f76b90ec46f708e1a2e1a47f13648ab9c0: Status 404 returned error can't find the container with id c426398600eb16c9b14d7884630c38f76b90ec46f708e1a2e1a47f13648ab9c0 Jul 07 19:02:06.341041 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:06.340954 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" event={"ID":"5b053333-9f76-412d-98c1-21a6e3a1d8e1","Type":"ContainerStarted","Data":"b7f8c2f3850504b4e0d9b01911df9c337c10fc0f0ec1fc66958040e8284a0ea0"} Jul 07 19:02:06.341041 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:06.341003 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" event={"ID":"5b053333-9f76-412d-98c1-21a6e3a1d8e1","Type":"ContainerStarted","Data":"c426398600eb16c9b14d7884630c38f76b90ec46f708e1a2e1a47f13648ab9c0"} Jul 07 19:02:06.361689 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:06.361647 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" podStartSLOduration=1.361634222 podStartE2EDuration="1.361634222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:06.361089435 +0000 UTC m=+2349.744882831" watchObservedRunningTime="2026-07-07 19:02:06.361634222 +0000 UTC m=+2349.745427615" Jul 07 19:02:07.197608 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:07.197576 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3d99b95-5ee8-486d-bd52-b3270cf72980" path="/var/lib/kubelet/pods/c3d99b95-5ee8-486d-bd52-b3270cf72980/volumes" Jul 07 19:02:09.650849 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:09.650813 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 19:02:16.063875 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:16.063844 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:16.064228 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:16.063887 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:16.076385 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:16.076362 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:16.403824 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:16.403741 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:02:21.748847 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:21.748816 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:26.041987 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:26.041952 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 19:02:47.742100 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:47.742060 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 19:02:57.221288 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:57.221257 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 19:02:57.227452 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:02:57.227427 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcq8x_1d688309-1ae2-40ac-9de5-3b37cf893695/ovn-acl-logging/0.log" Jul 07 19:03:00.651920 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:00.651882 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:03:00.750809 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:00.750772 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:03:05.861233 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:05.861193 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 19:03:12.564991 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:12.564955 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:03:12.664802 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:12.664773 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:03:13.021670 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:13.021629 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm"] Jul 07 19:03:13.110764 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:13.110714 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:03:14.096042 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:14.096007 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:03:23.035736 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:23.035658 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 19:03:27.033601 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.033572 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp"] Jul 07 19:03:27.037276 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.037249 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:27.044531 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.044506 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp"] Jul 07 19:03:27.051492 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.051472 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-972159\"/\"rate-test-model-972159-kserve-self-signed-certs\"" Jul 07 19:03:27.053056 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.053033 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:03:27.165560 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.165523 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-tmp-dir\") pod \"rate-test-model-972159-kserve-76cf7d87ff-nskdp\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:27.165755 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.165581 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-model-cache\") pod \"rate-test-model-972159-kserve-76cf7d87ff-nskdp\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:27.165755 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.165634 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-tls-certs\") pod \"rate-test-model-972159-kserve-76cf7d87ff-nskdp\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:27.165755 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.165678 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-home\") pod \"rate-test-model-972159-kserve-76cf7d87ff-nskdp\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:27.165755 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.165738 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-dshm\") pod \"rate-test-model-972159-kserve-76cf7d87ff-nskdp\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:27.165914 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.165758 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w9g2\" (UniqueName: \"kubernetes.io/projected/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-kube-api-access-9w9g2\") pod \"rate-test-model-972159-kserve-76cf7d87ff-nskdp\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:27.266790 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.266748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-tmp-dir\") pod \"rate-test-model-972159-kserve-76cf7d87ff-nskdp\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:27.267050 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.266808 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-model-cache\") pod \"rate-test-model-972159-kserve-76cf7d87ff-nskdp\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:27.267050 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.266842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-tls-certs\") pod \"rate-test-model-972159-kserve-76cf7d87ff-nskdp\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:27.267050 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.266861 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-home\") pod \"rate-test-model-972159-kserve-76cf7d87ff-nskdp\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:27.267050 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.266898 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-dshm\") pod \"rate-test-model-972159-kserve-76cf7d87ff-nskdp\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:27.267050 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.266922 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w9g2\" (UniqueName: \"kubernetes.io/projected/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-kube-api-access-9w9g2\") pod \"rate-test-model-972159-kserve-76cf7d87ff-nskdp\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:27.267401 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.267237 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-tmp-dir\") pod \"rate-test-model-972159-kserve-76cf7d87ff-nskdp\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:27.267401 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.267361 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-home\") pod \"rate-test-model-972159-kserve-76cf7d87ff-nskdp\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:27.267401 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.267386 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-model-cache\") pod \"rate-test-model-972159-kserve-76cf7d87ff-nskdp\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:27.269266 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.269226 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-dshm\") pod \"rate-test-model-972159-kserve-76cf7d87ff-nskdp\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:27.269424 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.269396 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-tls-certs\") pod \"rate-test-model-972159-kserve-76cf7d87ff-nskdp\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:27.276650 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.276629 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w9g2\" (UniqueName: \"kubernetes.io/projected/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-kube-api-access-9w9g2\") pod \"rate-test-model-972159-kserve-76cf7d87ff-nskdp\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:27.348369 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.348256 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:27.478620 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.478596 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp"] Jul 07 19:03:27.480668 ip-10-0-141-108 kubenswrapper[2578]: W0707 19:03:27.480638 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc55d347c_d2f3_48ef_83ba_ee3af4d21f48.slice/crio-264d3c552540764de1ac5969abde07a574e3db45aaf02398bef39c838d38d421 WatchSource:0}: Error finding container 264d3c552540764de1ac5969abde07a574e3db45aaf02398bef39c838d38d421: Status 404 returned error can't find the container with id 264d3c552540764de1ac5969abde07a574e3db45aaf02398bef39c838d38d421 Jul 07 19:03:27.733500 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.733465 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" event={"ID":"c55d347c-d2f3-48ef-83ba-ee3af4d21f48","Type":"ContainerStarted","Data":"ae6990f36d29870dcef3c9f19c82b39d4741f67caa418382c6947936e843a9d8"} Jul 07 19:03:27.733671 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.733512 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" event={"ID":"c55d347c-d2f3-48ef-83ba-ee3af4d21f48","Type":"ContainerStarted","Data":"264d3c552540764de1ac5969abde07a574e3db45aaf02398bef39c838d38d421"} Jul 07 19:03:27.750933 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:27.750888 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" podStartSLOduration=0.75087137 podStartE2EDuration="750.87137ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:03:27.749863857 +0000 UTC m=+2431.133657252" watchObservedRunningTime="2026-07-07 19:03:27.75087137 +0000 UTC m=+2431.134664765" Jul 07 19:03:28.041469 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.041372 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" podUID="5b053333-9f76-412d-98c1-21a6e3a1d8e1" containerName="main" containerID="cri-o://b7f8c2f3850504b4e0d9b01911df9c337c10fc0f0ec1fc66958040e8284a0ea0" gracePeriod=45 Jul 07 19:03:28.302229 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.302153 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:03:28.377880 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.377831 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-model-cache\") pod \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " Jul 07 19:03:28.377880 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.377885 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-home\") pod \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " Jul 07 19:03:28.378164 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.377907 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tvqh\" (UniqueName: \"kubernetes.io/projected/5b053333-9f76-412d-98c1-21a6e3a1d8e1-kube-api-access-4tvqh\") pod \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " Jul 07 19:03:28.378164 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.377936 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-tmp-dir\") pod \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " Jul 07 19:03:28.378164 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.377963 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-dshm\") pod \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " Jul 07 19:03:28.378164 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.378001 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b053333-9f76-412d-98c1-21a6e3a1d8e1-tls-certs\") pod \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\" (UID: \"5b053333-9f76-412d-98c1-21a6e3a1d8e1\") " Jul 07 19:03:28.378424 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.378165 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-home" (OuterVolumeSpecName: "home") pod "5b053333-9f76-412d-98c1-21a6e3a1d8e1" (UID: "5b053333-9f76-412d-98c1-21a6e3a1d8e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:03:28.378424 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.378183 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b053333-9f76-412d-98c1-21a6e3a1d8e1" (UID: "5b053333-9f76-412d-98c1-21a6e3a1d8e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:03:28.378424 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.378256 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b053333-9f76-412d-98c1-21a6e3a1d8e1" (UID: "5b053333-9f76-412d-98c1-21a6e3a1d8e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:03:28.378536 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.378429 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-model-cache\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:03:28.378536 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.378448 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-home\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:03:28.378536 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.378460 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-tmp-dir\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:03:28.380135 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.380116 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b053333-9f76-412d-98c1-21a6e3a1d8e1-kube-api-access-4tvqh" (OuterVolumeSpecName: "kube-api-access-4tvqh") pod "5b053333-9f76-412d-98c1-21a6e3a1d8e1" (UID: "5b053333-9f76-412d-98c1-21a6e3a1d8e1"). InnerVolumeSpecName "kube-api-access-4tvqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:03:28.380275 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.380258 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-dshm" (OuterVolumeSpecName: "dshm") pod "5b053333-9f76-412d-98c1-21a6e3a1d8e1" (UID: "5b053333-9f76-412d-98c1-21a6e3a1d8e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:03:28.380518 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.380500 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b053333-9f76-412d-98c1-21a6e3a1d8e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b053333-9f76-412d-98c1-21a6e3a1d8e1" (UID: "5b053333-9f76-412d-98c1-21a6e3a1d8e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:03:28.479158 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.479121 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b053333-9f76-412d-98c1-21a6e3a1d8e1-dshm\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:03:28.479158 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.479150 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b053333-9f76-412d-98c1-21a6e3a1d8e1-tls-certs\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:03:28.479158 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.479160 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tvqh\" (UniqueName: \"kubernetes.io/projected/5b053333-9f76-412d-98c1-21a6e3a1d8e1-kube-api-access-4tvqh\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:03:28.740940 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.740905 2578 generic.go:356] "Generic (PLEG): container finished" podID="5b053333-9f76-412d-98c1-21a6e3a1d8e1" containerID="b7f8c2f3850504b4e0d9b01911df9c337c10fc0f0ec1fc66958040e8284a0ea0" exitCode=0 Jul 07 19:03:28.741108 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.740955 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" Jul 07 19:03:28.741108 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.740988 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" event={"ID":"5b053333-9f76-412d-98c1-21a6e3a1d8e1","Type":"ContainerDied","Data":"b7f8c2f3850504b4e0d9b01911df9c337c10fc0f0ec1fc66958040e8284a0ea0"} Jul 07 19:03:28.741108 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.741031 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm" event={"ID":"5b053333-9f76-412d-98c1-21a6e3a1d8e1","Type":"ContainerDied","Data":"c426398600eb16c9b14d7884630c38f76b90ec46f708e1a2e1a47f13648ab9c0"} Jul 07 19:03:28.741108 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.741048 2578 scope.go:117] "RemoveContainer" containerID="b7f8c2f3850504b4e0d9b01911df9c337c10fc0f0ec1fc66958040e8284a0ea0" Jul 07 19:03:28.749644 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.749615 2578 scope.go:117] "RemoveContainer" containerID="b7f8c2f3850504b4e0d9b01911df9c337c10fc0f0ec1fc66958040e8284a0ea0" Jul 07 19:03:28.749867 ip-10-0-141-108 kubenswrapper[2578]: E0707 19:03:28.749847 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7f8c2f3850504b4e0d9b01911df9c337c10fc0f0ec1fc66958040e8284a0ea0\": container with ID starting with b7f8c2f3850504b4e0d9b01911df9c337c10fc0f0ec1fc66958040e8284a0ea0 not found: ID does not exist" containerID="b7f8c2f3850504b4e0d9b01911df9c337c10fc0f0ec1fc66958040e8284a0ea0" Jul 07 19:03:28.749932 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.749884 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7f8c2f3850504b4e0d9b01911df9c337c10fc0f0ec1fc66958040e8284a0ea0"} err="failed to get container status \"b7f8c2f3850504b4e0d9b01911df9c337c10fc0f0ec1fc66958040e8284a0ea0\": rpc error: code = NotFound desc = could not find container \"b7f8c2f3850504b4e0d9b01911df9c337c10fc0f0ec1fc66958040e8284a0ea0\": container with ID starting with b7f8c2f3850504b4e0d9b01911df9c337c10fc0f0ec1fc66958040e8284a0ea0 not found: ID does not exist" Jul 07 19:03:28.764440 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.764414 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm"] Jul 07 19:03:28.769145 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:28.769122 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-972159/sub-test-model-972159-kserve-774cc75545-hjtgm"] Jul 07 19:03:29.197818 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:29.197727 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b053333-9f76-412d-98c1-21a6e3a1d8e1" path="/var/lib/kubelet/pods/5b053333-9f76-412d-98c1-21a6e3a1d8e1/volumes" Jul 07 19:03:37.348901 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:37.348845 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:37.348901 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:37.348911 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:37.361707 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:37.361671 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:37.796665 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:37.796635 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:03:39.442452 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:39.442409 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 19:03:40.915504 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:40.915454 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:03:56.595930 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:03:56.595895 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:04:00.735568 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:00.735531 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 19:04:19.378285 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:19.378248 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 19:04:33.743816 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:33.743775 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 19:04:41.209369 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:41.209334 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp"] Jul 07 19:04:41.232266 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:41.232233 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:04:42.073505 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:42.073473 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:04:42.328473 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:42.328387 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:04:42.436474 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:42.436443 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:04:43.290303 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:43.290272 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:04:43.889717 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:43.889685 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:04:44.069763 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:44.069726 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:04:46.855160 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:46.855075 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 19:04:48.865194 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:48.865130 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:04:50.959621 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:50.959589 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:04:51.061791 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:51.061757 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:04:56.228996 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:56.228949 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" podUID="c55d347c-d2f3-48ef-83ba-ee3af4d21f48" containerName="main" containerID="cri-o://ae6990f36d29870dcef3c9f19c82b39d4741f67caa418382c6947936e843a9d8" gracePeriod=45 Jul 07 19:04:56.482113 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:56.482037 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:04:56.589637 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:56.589588 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-dshm\") pod \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " Jul 07 19:04:56.589856 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:56.589650 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9w9g2\" (UniqueName: \"kubernetes.io/projected/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-kube-api-access-9w9g2\") pod \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " Jul 07 19:04:56.589856 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:56.589681 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-tmp-dir\") pod \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " Jul 07 19:04:56.589856 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:56.589711 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-tls-certs\") pod \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " Jul 07 19:04:56.589856 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:56.589777 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-model-cache\") pod \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " Jul 07 19:04:56.589856 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:56.589828 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-home\") pod \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\" (UID: \"c55d347c-d2f3-48ef-83ba-ee3af4d21f48\") " Jul 07 19:04:56.590153 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:56.590014 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c55d347c-d2f3-48ef-83ba-ee3af4d21f48" (UID: "c55d347c-d2f3-48ef-83ba-ee3af4d21f48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:04:56.590212 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:56.590150 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-model-cache" (OuterVolumeSpecName: "model-cache") pod "c55d347c-d2f3-48ef-83ba-ee3af4d21f48" (UID: "c55d347c-d2f3-48ef-83ba-ee3af4d21f48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:04:56.590274 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:56.590186 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-tmp-dir\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:04:56.590274 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:56.590222 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-home" (OuterVolumeSpecName: "home") pod "c55d347c-d2f3-48ef-83ba-ee3af4d21f48" (UID: "c55d347c-d2f3-48ef-83ba-ee3af4d21f48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:04:56.591985 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:56.591958 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-dshm" (OuterVolumeSpecName: "dshm") pod "c55d347c-d2f3-48ef-83ba-ee3af4d21f48" (UID: "c55d347c-d2f3-48ef-83ba-ee3af4d21f48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:04:56.591985 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:56.591954 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-kube-api-access-9w9g2" (OuterVolumeSpecName: "kube-api-access-9w9g2") pod "c55d347c-d2f3-48ef-83ba-ee3af4d21f48" (UID: "c55d347c-d2f3-48ef-83ba-ee3af4d21f48"). InnerVolumeSpecName "kube-api-access-9w9g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:04:56.592311 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:56.592289 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c55d347c-d2f3-48ef-83ba-ee3af4d21f48" (UID: "c55d347c-d2f3-48ef-83ba-ee3af4d21f48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:04:56.691128 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:56.691091 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-dshm\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:04:56.691128 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:56.691123 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w9g2\" (UniqueName: \"kubernetes.io/projected/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-kube-api-access-9w9g2\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:04:56.691128 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:56.691134 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-tls-certs\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:04:56.691419 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:56.691143 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-model-cache\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:04:56.691419 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:56.691151 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c55d347c-d2f3-48ef-83ba-ee3af4d21f48-home\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:04:57.159591 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:57.159561 2578 generic.go:356] "Generic (PLEG): container finished" podID="c55d347c-d2f3-48ef-83ba-ee3af4d21f48" containerID="ae6990f36d29870dcef3c9f19c82b39d4741f67caa418382c6947936e843a9d8" exitCode=0 Jul 07 19:04:57.159766 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:57.159607 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" Jul 07 19:04:57.159766 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:57.159637 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" event={"ID":"c55d347c-d2f3-48ef-83ba-ee3af4d21f48","Type":"ContainerDied","Data":"ae6990f36d29870dcef3c9f19c82b39d4741f67caa418382c6947936e843a9d8"} Jul 07 19:04:57.159766 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:57.159678 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp" event={"ID":"c55d347c-d2f3-48ef-83ba-ee3af4d21f48","Type":"ContainerDied","Data":"264d3c552540764de1ac5969abde07a574e3db45aaf02398bef39c838d38d421"} Jul 07 19:04:57.159766 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:57.159695 2578 scope.go:117] "RemoveContainer" containerID="ae6990f36d29870dcef3c9f19c82b39d4741f67caa418382c6947936e843a9d8" Jul 07 19:04:57.168829 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:57.168810 2578 scope.go:117] "RemoveContainer" containerID="ae6990f36d29870dcef3c9f19c82b39d4741f67caa418382c6947936e843a9d8" Jul 07 19:04:57.169069 ip-10-0-141-108 kubenswrapper[2578]: E0707 19:04:57.169050 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae6990f36d29870dcef3c9f19c82b39d4741f67caa418382c6947936e843a9d8\": container with ID starting with ae6990f36d29870dcef3c9f19c82b39d4741f67caa418382c6947936e843a9d8 not found: ID does not exist" containerID="ae6990f36d29870dcef3c9f19c82b39d4741f67caa418382c6947936e843a9d8" Jul 07 19:04:57.169135 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:57.169082 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6990f36d29870dcef3c9f19c82b39d4741f67caa418382c6947936e843a9d8"} err="failed to get container status \"ae6990f36d29870dcef3c9f19c82b39d4741f67caa418382c6947936e843a9d8\": rpc error: code = NotFound desc = could not find container \"ae6990f36d29870dcef3c9f19c82b39d4741f67caa418382c6947936e843a9d8\": container with ID starting with ae6990f36d29870dcef3c9f19c82b39d4741f67caa418382c6947936e843a9d8 not found: ID does not exist" Jul 07 19:04:57.201288 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:57.201258 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp"] Jul 07 19:04:57.205620 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:57.205594 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-972159/rate-test-model-972159-kserve-76cf7d87ff-nskdp"] Jul 07 19:04:59.198014 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:04:59.197931 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c55d347c-d2f3-48ef-83ba-ee3af4d21f48" path="/var/lib/kubelet/pods/c55d347c-d2f3-48ef-83ba-ee3af4d21f48/volumes" Jul 07 19:05:26.138023 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:26.137962 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:26.360548 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:26.360511 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2"] Jul 07 19:05:26.360780 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:26.360748 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" podUID="cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" containerName="istio-proxy" containerID="cri-o://67629322e27949f773cdb750014f8e882ba2b8a821837261c183e6235f2ee0cf" gracePeriod=30 Jul 07 19:05:26.395241 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:26.395151 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:26.603253 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:26.603218 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:26.800277 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:26.800241 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:27.231435 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:27.231403 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:27.304674 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:27.304636 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:29.237073 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:29.237040 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-74l2w"] Jul 07 19:05:31.618308 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.618283 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 19:05:31.684073 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.683976 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-credential-socket\") pod \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " Jul 07 19:05:31.684073 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.684032 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-token\") pod \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " Jul 07 19:05:31.684340 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.684078 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istiod-ca-cert\") pod \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " Jul 07 19:05:31.684340 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.684109 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-workload-certs\") pod \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " Jul 07 19:05:31.684340 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.684162 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-data\") pod \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " Jul 07 19:05:31.684340 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.684190 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-envoy\") pod \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " Jul 07 19:05:31.684340 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.684243 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwhbg\" (UniqueName: \"kubernetes.io/projected/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-kube-api-access-gwhbg\") pod \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " Jul 07 19:05:31.684340 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.684287 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-workload-socket\") pod \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " Jul 07 19:05:31.684340 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.684286 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" (UID: "cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:05:31.684713 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.684358 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-podinfo\") pod \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\" (UID: \"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2\") " Jul 07 19:05:31.684713 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.684625 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" (UID: "cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:05:31.684820 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.684718 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-credential-socket\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:05:31.684820 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.684739 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istiod-ca-cert\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:05:31.685122 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.685087 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" (UID: "cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:05:31.685250 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.685144 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-data" (OuterVolumeSpecName: "istio-data") pod "cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" (UID: "cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:05:31.685550 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.685520 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" (UID: "cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:05:31.686619 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.686590 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" (UID: "cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:05:31.686719 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.686646 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-token" (OuterVolumeSpecName: "istio-token") pod "cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" (UID: "cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:05:31.686899 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.686862 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-kube-api-access-gwhbg" (OuterVolumeSpecName: "kube-api-access-gwhbg") pod "cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" (UID: "cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2"). InnerVolumeSpecName "kube-api-access-gwhbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:05:31.686899 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.686886 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" (UID: "cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:05:31.785976 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.785936 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-token\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:05:31.785976 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.785963 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-workload-certs\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:05:31.785976 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.785974 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-data\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:05:31.785976 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.785982 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-envoy\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:05:31.786246 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.785990 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwhbg\" (UniqueName: \"kubernetes.io/projected/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-kube-api-access-gwhbg\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:05:31.786246 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.786000 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-workload-socket\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:05:31.786246 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:31.786010 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2-istio-podinfo\") on node \"ip-10-0-141-108.ec2.internal\" DevicePath \"\"" Jul 07 19:05:32.327657 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:32.327621 2578 generic.go:356] "Generic (PLEG): container finished" podID="cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" containerID="67629322e27949f773cdb750014f8e882ba2b8a821837261c183e6235f2ee0cf" exitCode=0 Jul 07 19:05:32.327836 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:32.327665 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" Jul 07 19:05:32.327836 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:32.327710 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" event={"ID":"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2","Type":"ContainerDied","Data":"67629322e27949f773cdb750014f8e882ba2b8a821837261c183e6235f2ee0cf"} Jul 07 19:05:32.327836 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:32.327771 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2" event={"ID":"cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2","Type":"ContainerDied","Data":"6c6daed4d27d4f95e4ea3587b88c923898ad9af2971f39b92ecd286365223e8e"} Jul 07 19:05:32.327836 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:32.327793 2578 scope.go:117] "RemoveContainer" containerID="67629322e27949f773cdb750014f8e882ba2b8a821837261c183e6235f2ee0cf" Jul 07 19:05:32.337505 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:32.337486 2578 scope.go:117] "RemoveContainer" containerID="288f1a74bf1074f995e10d82428d9c9c7c82a1f14e73e7e5645e2a59ee4785d7" Jul 07 19:05:32.345469 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:32.345449 2578 scope.go:117] "RemoveContainer" containerID="67629322e27949f773cdb750014f8e882ba2b8a821837261c183e6235f2ee0cf" Jul 07 19:05:32.345730 ip-10-0-141-108 kubenswrapper[2578]: E0707 19:05:32.345709 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67629322e27949f773cdb750014f8e882ba2b8a821837261c183e6235f2ee0cf\": container with ID starting with 67629322e27949f773cdb750014f8e882ba2b8a821837261c183e6235f2ee0cf not found: ID does not exist" containerID="67629322e27949f773cdb750014f8e882ba2b8a821837261c183e6235f2ee0cf" Jul 07 19:05:32.345781 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:32.345739 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67629322e27949f773cdb750014f8e882ba2b8a821837261c183e6235f2ee0cf"} err="failed to get container status \"67629322e27949f773cdb750014f8e882ba2b8a821837261c183e6235f2ee0cf\": rpc error: code = NotFound desc = could not find container \"67629322e27949f773cdb750014f8e882ba2b8a821837261c183e6235f2ee0cf\": container with ID starting with 67629322e27949f773cdb750014f8e882ba2b8a821837261c183e6235f2ee0cf not found: ID does not exist" Jul 07 19:05:32.345781 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:32.345757 2578 scope.go:117] "RemoveContainer" containerID="288f1a74bf1074f995e10d82428d9c9c7c82a1f14e73e7e5645e2a59ee4785d7" Jul 07 19:05:32.346017 ip-10-0-141-108 kubenswrapper[2578]: E0707 19:05:32.345998 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288f1a74bf1074f995e10d82428d9c9c7c82a1f14e73e7e5645e2a59ee4785d7\": container with ID starting with 288f1a74bf1074f995e10d82428d9c9c7c82a1f14e73e7e5645e2a59ee4785d7 not found: ID does not exist" containerID="288f1a74bf1074f995e10d82428d9c9c7c82a1f14e73e7e5645e2a59ee4785d7" Jul 07 19:05:32.346084 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:32.346022 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288f1a74bf1074f995e10d82428d9c9c7c82a1f14e73e7e5645e2a59ee4785d7"} err="failed to get container status \"288f1a74bf1074f995e10d82428d9c9c7c82a1f14e73e7e5645e2a59ee4785d7\": rpc error: code = NotFound desc = could not find container \"288f1a74bf1074f995e10d82428d9c9c7c82a1f14e73e7e5645e2a59ee4785d7\": container with ID starting with 288f1a74bf1074f995e10d82428d9c9c7c82a1f14e73e7e5645e2a59ee4785d7 not found: ID does not exist" Jul 07 19:05:32.354871 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:32.354844 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2"] Jul 07 19:05:32.356472 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:32.356453 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-972159-openshift-default-568589b9cf-dsbz2"] Jul 07 19:05:33.197551 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:33.197519 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" path="/var/lib/kubelet/pods/cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2/volumes" Jul 07 19:05:33.891370 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:33.891310 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:33.974640 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:33.974605 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:34.222034 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:34.222001 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:34.519638 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:34.519540 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:40.779405 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:40.779372 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-hkzl2_018abded-bab5-4066-803b-f387501cc678/manager/0.log" Jul 07 19:05:40.911742 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:40.911699 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-v2lrb_fa343ac0-bade-49dd-9fc5-35778b5e2ee7/manager/0.log" Jul 07 19:05:41.554445 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:41.554416 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-llh67_2f3548c4-1727-409f-88f2-2301a2ad2204/server/0.log" Jul 07 19:05:41.682405 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:41.682378 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-487nt_d5c83c3f-318d-4c87-ace1-01608156d78b/manager/0.log" Jul 07 19:05:41.810515 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:41.810436 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7595c859c8-vk6nq_1b4458c2-e0f3-4be4-9691-58e3f4f42d38/copy-manifests/0.log" Jul 07 19:05:41.823793 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:41.823761 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7595c859c8-vk6nq_1b4458c2-e0f3-4be4-9691-58e3f4f42d38/manager/0.log" Jul 07 19:05:43.715196 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:43.715165 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-96qch_71414372-dc1a-42f8-8711-e24686b4f505/manager/0.log" Jul 07 19:05:43.839368 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:43.839306 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-pcjlp_67cece42-5d7a-41d1-adf0-10579cdb66bb/manager/0.log" Jul 07 19:05:44.067548 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:44.067514 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-mjf79_4b300561-22d7-4581-a149-ee9847db2eb3/registry-server/0.log" Jul 07 19:05:44.197448 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:44.197409 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-lsfhk_071b8ef2-d40c-40e5-8f28-3bfa998f1bd1/manager/0.log" Jul 07 19:05:44.324987 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:44.324917 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-74l2w_8d18b117-f206-4c2c-a9db-432663ab515d/limitador/0.log" Jul 07 19:05:44.455607 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:44.455571 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-9f846_12a67f8c-909f-49c0-960f-2f8c51692968/manager/0.log" Jul 07 19:05:45.062141 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:45.062110 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f5865f688-kgtgc_73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107/kube-auth-proxy/0.log" Jul 07 19:05:45.308107 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:45.308074 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-65gv5_b39b4b49-06c6-4309-ac4e-ca8c140a9a2d/istio-proxy/1.log" Jul 07 19:05:46.548165 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:46.548131 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-qwn87_7df3f97a-0327-48e6-862d-8bc4d58fc99c/main/0.log" Jul 07 19:05:46.666093 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:46.666038 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-w44w7_130c95ce-3eff-4c85-a5de-719730e28566/main/0.log" Jul 07 19:05:51.334273 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.334222 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fzth9/must-gather-9d6rz"] Jul 07 19:05:51.334808 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.334732 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b053333-9f76-412d-98c1-21a6e3a1d8e1" containerName="main" Jul 07 19:05:51.334808 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.334751 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b053333-9f76-412d-98c1-21a6e3a1d8e1" containerName="main" Jul 07 19:05:51.334808 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.334766 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" containerName="istio-proxy" Jul 07 19:05:51.334808 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.334774 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" containerName="istio-proxy" Jul 07 19:05:51.334808 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.334788 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c55d347c-d2f3-48ef-83ba-ee3af4d21f48" containerName="main" Jul 07 19:05:51.334808 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.334796 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c55d347c-d2f3-48ef-83ba-ee3af4d21f48" containerName="main" Jul 07 19:05:51.334808 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.334808 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" containerName="istio-proxy" Jul 07 19:05:51.335173 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.334816 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" containerName="istio-proxy" Jul 07 19:05:51.335173 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.334913 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" containerName="istio-proxy" Jul 07 19:05:51.335173 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.334926 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc5b7e47-9f2b-4200-aa8f-8cbf06ba08c2" containerName="istio-proxy" Jul 07 19:05:51.335173 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.334938 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c55d347c-d2f3-48ef-83ba-ee3af4d21f48" containerName="main" Jul 07 19:05:51.335173 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.334948 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b053333-9f76-412d-98c1-21a6e3a1d8e1" containerName="main" Jul 07 19:05:51.339728 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.339702 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fzth9/must-gather-9d6rz" Jul 07 19:05:51.342534 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.342510 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fzth9\"/\"kube-root-ca.crt\"" Jul 07 19:05:51.342898 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.342872 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fzth9\"/\"openshift-service-ca.crt\"" Jul 07 19:05:51.343011 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.342916 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fzth9\"/\"default-dockercfg-fmmcn\"" Jul 07 19:05:51.350927 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.350901 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fzth9/must-gather-9d6rz"] Jul 07 19:05:51.356263 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.356235 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:51.454183 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.454145 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:51.462420 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.462394 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/118a0099-8e8e-4752-823d-c99af5400666-must-gather-output\") pod \"must-gather-9d6rz\" (UID: \"118a0099-8e8e-4752-823d-c99af5400666\") " pod="openshift-must-gather-fzth9/must-gather-9d6rz" Jul 07 19:05:51.462574 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.462488 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqph8\" (UniqueName: \"kubernetes.io/projected/118a0099-8e8e-4752-823d-c99af5400666-kube-api-access-vqph8\") pod \"must-gather-9d6rz\" (UID: \"118a0099-8e8e-4752-823d-c99af5400666\") " pod="openshift-must-gather-fzth9/must-gather-9d6rz" Jul 07 19:05:51.563780 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.563740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqph8\" (UniqueName: \"kubernetes.io/projected/118a0099-8e8e-4752-823d-c99af5400666-kube-api-access-vqph8\") pod \"must-gather-9d6rz\" (UID: \"118a0099-8e8e-4752-823d-c99af5400666\") " pod="openshift-must-gather-fzth9/must-gather-9d6rz" Jul 07 19:05:51.563966 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.563799 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/118a0099-8e8e-4752-823d-c99af5400666-must-gather-output\") pod \"must-gather-9d6rz\" (UID: \"118a0099-8e8e-4752-823d-c99af5400666\") " pod="openshift-must-gather-fzth9/must-gather-9d6rz" Jul 07 19:05:51.564115 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.564099 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/118a0099-8e8e-4752-823d-c99af5400666-must-gather-output\") pod \"must-gather-9d6rz\" (UID: \"118a0099-8e8e-4752-823d-c99af5400666\") " pod="openshift-must-gather-fzth9/must-gather-9d6rz" Jul 07 19:05:51.574348 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.574303 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqph8\" (UniqueName: \"kubernetes.io/projected/118a0099-8e8e-4752-823d-c99af5400666-kube-api-access-vqph8\") pod \"must-gather-9d6rz\" (UID: \"118a0099-8e8e-4752-823d-c99af5400666\") " pod="openshift-must-gather-fzth9/must-gather-9d6rz" Jul 07 19:05:51.649438 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.649316 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fzth9/must-gather-9d6rz" Jul 07 19:05:51.787536 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.787511 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fzth9/must-gather-9d6rz"] Jul 07 19:05:51.789568 ip-10-0-141-108 kubenswrapper[2578]: W0707 19:05:51.789535 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod118a0099_8e8e_4752_823d_c99af5400666.slice/crio-bc0afcec06fc5242c68d88e2d3defc0da0b5827cbbf53cc67644a69278308fb8 WatchSource:0}: Error finding container bc0afcec06fc5242c68d88e2d3defc0da0b5827cbbf53cc67644a69278308fb8: Status 404 returned error can't find the container with id bc0afcec06fc5242c68d88e2d3defc0da0b5827cbbf53cc67644a69278308fb8 Jul 07 19:05:51.791420 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:51.791387 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:05:52.417332 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:52.417282 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fzth9/must-gather-9d6rz" event={"ID":"118a0099-8e8e-4752-823d-c99af5400666","Type":"ContainerStarted","Data":"bc0afcec06fc5242c68d88e2d3defc0da0b5827cbbf53cc67644a69278308fb8"} Jul 07 19:05:53.425079 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:53.425031 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fzth9/must-gather-9d6rz" event={"ID":"118a0099-8e8e-4752-823d-c99af5400666","Type":"ContainerStarted","Data":"90150bd7fb502afffc960635565a537787026e3ac7599ef3801fdaad8b7032aa"} Jul 07 19:05:53.425613 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:53.425094 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fzth9/must-gather-9d6rz" event={"ID":"118a0099-8e8e-4752-823d-c99af5400666","Type":"ContainerStarted","Data":"94f5834aa527a5f1a968472f45ccd68364a9dd165a2f1d4897bace7ae15a8542"} Jul 07 19:05:53.444700 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:53.444628 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fzth9/must-gather-9d6rz" podStartSLOduration=1.347718107 podStartE2EDuration="2.444603992s" totalImagesPullingTime="1.096885885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:05:53.441309926 +0000 UTC m=+2576.825103322" watchObservedRunningTime="2026-07-07 19:05:53.444603992 +0000 UTC m=+2576.828397388" Jul 07 19:05:54.662455 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:54.662422 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-98qb4_e17ddfe4-73b3-45bd-b80f-ed992e7e1b71/global-pull-secret-syncer/0.log" Jul 07 19:05:54.817716 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:54.817681 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" Jul 07 19:05:54.951563 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:05:54.951480 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-108.ec2.internal_f2b9e46c04925be656b0777b331f09d9/haproxy/0.log" Jul 07 19:06:00.207434 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:00.207380 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-96qch_71414372-dc1a-42f8-8711-e24686b4f505/manager/0.log" Jul 07 19:06:00.237573 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:00.237533 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-pcjlp_67cece42-5d7a-41d1-adf0-10579cdb66bb/manager/0.log" Jul 07 19:06:00.326084 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:00.326032 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-mjf79_4b300561-22d7-4581-a149-ee9847db2eb3/registry-server/0.log" Jul 07 19:06:00.499357 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:00.498965 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-lsfhk_071b8ef2-d40c-40e5-8f28-3bfa998f1bd1/manager/0.log" Jul 07 19:06:00.549889 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:00.549845 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-74l2w_8d18b117-f206-4c2c-a9db-432663ab515d/limitador/0.log" Jul 07 19:06:00.679345 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:00.678980 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-9f846_12a67f8c-909f-49c0-960f-2f8c51692968/manager/0.log" Jul 07 19:06:01.672982 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:01.672917 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:02.853692 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:02.853660 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gxdbb_d0acf142-00b0-44ca-a46d-7d8e3f86f841/node-exporter/0.log" Jul 07 19:06:02.881898 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:02.881850 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gxdbb_d0acf142-00b0-44ca-a46d-7d8e3f86f841/kube-rbac-proxy/0.log" Jul 07 19:06:02.906491 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:02.906452 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gxdbb_d0acf142-00b0-44ca-a46d-7d8e3f86f841/init-textfile/0.log" Jul 07 19:06:05.009348 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:05.009283 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm"] Jul 07 19:06:05.024654 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:05.024615 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" Jul 07 19:06:05.029052 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:05.029019 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm"] Jul 07 19:06:05.102818 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:05.102773 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ab2ae7b-ab80-4415-aeee-7538c1e31ed0-lib-modules\") pod \"perf-node-gather-daemonset-76hsm\" (UID: \"6ab2ae7b-ab80-4415-aeee-7538c1e31ed0\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" Jul 07 19:06:05.103026 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:05.102847 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ab2ae7b-ab80-4415-aeee-7538c1e31ed0-podres\") pod \"perf-node-gather-daemonset-76hsm\" (UID: \"6ab2ae7b-ab80-4415-aeee-7538c1e31ed0\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" Jul 07 19:06:05.103026 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:05.102897 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ab2ae7b-ab80-4415-aeee-7538c1e31ed0-proc\") pod \"perf-node-gather-daemonset-76hsm\" (UID: \"6ab2ae7b-ab80-4415-aeee-7538c1e31ed0\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" Jul 07 19:06:05.103026 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:05.102927 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfl58\" (UniqueName: \"kubernetes.io/projected/6ab2ae7b-ab80-4415-aeee-7538c1e31ed0-kube-api-access-nfl58\") pod \"perf-node-gather-daemonset-76hsm\" (UID: \"6ab2ae7b-ab80-4415-aeee-7538c1e31ed0\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" Jul 07 19:06:05.103026 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:05.102969 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ab2ae7b-ab80-4415-aeee-7538c1e31ed0-sys\") pod \"perf-node-gather-daemonset-76hsm\" (UID: \"6ab2ae7b-ab80-4415-aeee-7538c1e31ed0\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" Jul 07 19:06:05.203869 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:05.203821 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ab2ae7b-ab80-4415-aeee-7538c1e31ed0-lib-modules\") pod \"perf-node-gather-daemonset-76hsm\" (UID: \"6ab2ae7b-ab80-4415-aeee-7538c1e31ed0\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" Jul 07 19:06:05.204072 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:05.203883 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ab2ae7b-ab80-4415-aeee-7538c1e31ed0-podres\") pod \"perf-node-gather-daemonset-76hsm\" (UID: \"6ab2ae7b-ab80-4415-aeee-7538c1e31ed0\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" Jul 07 19:06:05.204072 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:05.203929 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ab2ae7b-ab80-4415-aeee-7538c1e31ed0-proc\") pod \"perf-node-gather-daemonset-76hsm\" (UID: \"6ab2ae7b-ab80-4415-aeee-7538c1e31ed0\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" Jul 07 19:06:05.204072 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:05.203958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfl58\" (UniqueName: \"kubernetes.io/projected/6ab2ae7b-ab80-4415-aeee-7538c1e31ed0-kube-api-access-nfl58\") pod \"perf-node-gather-daemonset-76hsm\" (UID: \"6ab2ae7b-ab80-4415-aeee-7538c1e31ed0\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" Jul 07 19:06:05.204072 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:05.203993 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ab2ae7b-ab80-4415-aeee-7538c1e31ed0-sys\") pod \"perf-node-gather-daemonset-76hsm\" (UID: \"6ab2ae7b-ab80-4415-aeee-7538c1e31ed0\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" Jul 07 19:06:05.204072 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:05.204028 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ab2ae7b-ab80-4415-aeee-7538c1e31ed0-lib-modules\") pod \"perf-node-gather-daemonset-76hsm\" (UID: \"6ab2ae7b-ab80-4415-aeee-7538c1e31ed0\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" Jul 07 19:06:05.204072 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:05.204036 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ab2ae7b-ab80-4415-aeee-7538c1e31ed0-podres\") pod \"perf-node-gather-daemonset-76hsm\" (UID: \"6ab2ae7b-ab80-4415-aeee-7538c1e31ed0\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" Jul 07 19:06:05.204349 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:05.204030 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ab2ae7b-ab80-4415-aeee-7538c1e31ed0-proc\") pod \"perf-node-gather-daemonset-76hsm\" (UID: \"6ab2ae7b-ab80-4415-aeee-7538c1e31ed0\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" Jul 07 19:06:05.204349 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:05.204119 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ab2ae7b-ab80-4415-aeee-7538c1e31ed0-sys\") pod \"perf-node-gather-daemonset-76hsm\" (UID: \"6ab2ae7b-ab80-4415-aeee-7538c1e31ed0\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" Jul 07 19:06:05.228828 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:05.228650 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfl58\" (UniqueName: \"kubernetes.io/projected/6ab2ae7b-ab80-4415-aeee-7538c1e31ed0-kube-api-access-nfl58\") pod \"perf-node-gather-daemonset-76hsm\" (UID: \"6ab2ae7b-ab80-4415-aeee-7538c1e31ed0\") " pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" Jul 07 19:06:05.338465 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:05.338373 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" Jul 07 19:06:05.526182 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:05.525539 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm"] Jul 07 19:06:05.529419 ip-10-0-141-108 kubenswrapper[2578]: W0707 19:06:05.529371 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6ab2ae7b_ab80_4415_aeee_7538c1e31ed0.slice/crio-a815e2bec289d69ef68d4b00f4caaa5dcbd070c91eb9c34e341d56cb32505fbe WatchSource:0}: Error finding container a815e2bec289d69ef68d4b00f4caaa5dcbd070c91eb9c34e341d56cb32505fbe: Status 404 returned error can't find the container with id a815e2bec289d69ef68d4b00f4caaa5dcbd070c91eb9c34e341d56cb32505fbe Jul 07 19:06:06.084793 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:06.084743 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-57f5bb7b7f-8mfgl_2a7202c3-0e70-4409-a04c-6af1caa39305/console/0.log" Jul 07 19:06:06.523151 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:06.523099 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" event={"ID":"6ab2ae7b-ab80-4415-aeee-7538c1e31ed0","Type":"ContainerStarted","Data":"233f8cf12ff178766ead38dc52bbc12542a347eeb5d0d8024407858df099b4fe"} Jul 07 19:06:06.523399 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:06.523180 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" event={"ID":"6ab2ae7b-ab80-4415-aeee-7538c1e31ed0","Type":"ContainerStarted","Data":"a815e2bec289d69ef68d4b00f4caaa5dcbd070c91eb9c34e341d56cb32505fbe"} Jul 07 19:06:06.524026 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:06.523993 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" Jul 07 19:06:06.541160 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:06.541078 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" podStartSLOduration=2.541055634 podStartE2EDuration="2.541055634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:06:06.538471301 +0000 UTC m=+2589.922264697" watchObservedRunningTime="2026-07-07 19:06:06.541055634 +0000 UTC m=+2589.924849031" Jul 07 19:06:07.891949 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:07.891892 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bncsv_03509627-aac5-42cf-a76b-4b7e499bdb4e/dns/0.log" Jul 07 19:06:07.917803 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:07.917775 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bncsv_03509627-aac5-42cf-a76b-4b7e499bdb4e/kube-rbac-proxy/0.log" Jul 07 19:06:08.000779 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:08.000752 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bnkff_bf3379f6-bbce-4398-bb71-1dbe3ddc5059/dns-node-resolver/0.log" Jul 07 19:06:08.734608 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:08.734580 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wlm6k_d3f4cdc4-da50-4c22-8e51-55872deaf1c8/node-ca/0.log" Jul 07 19:06:09.877618 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:09.877579 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f5865f688-kgtgc_73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107/kube-auth-proxy/0.log" Jul 07 19:06:09.939811 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:09.939770 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-65gv5_b39b4b49-06c6-4309-ac4e-ca8c140a9a2d/istio-proxy/0.log" Jul 07 19:06:09.980642 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:09.980603 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-65gv5_b39b4b49-06c6-4309-ac4e-ca8c140a9a2d/istio-proxy/1.log" Jul 07 19:06:10.926541 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:10.926506 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sxzvf_be81195f-a6f0-4154-abde-ccaa50f68346/serve-healthcheck-canary/0.log" Jul 07 19:06:11.639510 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:11.639474 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pd2qx_5e5a23b5-a6ca-4693-afc4-489102504294/kube-rbac-proxy/0.log" Jul 07 19:06:11.668723 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:11.668698 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pd2qx_5e5a23b5-a6ca-4693-afc4-489102504294/exporter/0.log" Jul 07 19:06:11.694422 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:11.694394 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pd2qx_5e5a23b5-a6ca-4693-afc4-489102504294/extractor/0.log" Jul 07 19:06:13.549947 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:13.549911 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fzth9/perf-node-gather-daemonset-76hsm" Jul 07 19:06:13.983522 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:13.983496 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-hkzl2_018abded-bab5-4066-803b-f387501cc678/manager/0.log" Jul 07 19:06:14.174206 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:14.174163 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-v2lrb_fa343ac0-bade-49dd-9fc5-35778b5e2ee7/manager/0.log" Jul 07 19:06:14.466339 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:14.466252 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-llh67_2f3548c4-1727-409f-88f2-2301a2ad2204/server/0.log" Jul 07 19:06:14.557941 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:14.557900 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-487nt_d5c83c3f-318d-4c87-ace1-01608156d78b/manager/0.log" Jul 07 19:06:14.671727 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:14.671691 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7595c859c8-vk6nq_1b4458c2-e0f3-4be4-9691-58e3f4f42d38/manager/0.log" Jul 07 19:06:14.691267 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:14.691235 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7595c859c8-vk6nq_1b4458c2-e0f3-4be4-9691-58e3f4f42d38/copy-manifests/0.log" Jul 07 19:06:15.684023 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:15.683997 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f5865f688-kgtgc_73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107/kube-auth-proxy/0.log" Jul 07 19:06:15.737580 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:15.737550 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-65gv5_b39b4b49-06c6-4309-ac4e-ca8c140a9a2d/istio-proxy/0.log" Jul 07 19:06:15.779853 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:15.779820 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-65gv5_b39b4b49-06c6-4309-ac4e-ca8c140a9a2d/istio-proxy/1.log" Jul 07 19:06:17.243944 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:17.243915 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f5865f688-kgtgc_73c3ff7b-0cc1-4a70-bb0b-16d0c8b90107/kube-auth-proxy/0.log" Jul 07 19:06:17.296210 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:17.296160 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-65gv5_b39b4b49-06c6-4309-ac4e-ca8c140a9a2d/istio-proxy/0.log" Jul 07 19:06:17.334928 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:17.334893 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-65gv5_b39b4b49-06c6-4309-ac4e-ca8c140a9a2d/istio-proxy/1.log" Jul 07 19:06:17.663020 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:17.662908 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66cjm_88949732-a999-49ba-b1ba-3f283bbd1c45/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:23.346715 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:23.346684 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9m4cn_c9c36400-3fba-4f9a-9cf3-0d8cf8ebe448/kube-multus/0.log" Jul 07 19:06:23.845004 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:23.844976 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wgd4k_f1e7507b-7367-4253-bb48-212f27b5f8ca/kube-multus-additional-cni-plugins/0.log" Jul 07 19:06:23.883727 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:23.883694 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wgd4k_f1e7507b-7367-4253-bb48-212f27b5f8ca/egress-router-binary-copy/0.log" Jul 07 19:06:23.908814 ip-10-0-141-108 kubenswrapper[2578]: I0707 19:06:23.908780 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wgd4k_f1e7507b-7367-4253-bb48-212f27b5f8ca/cni-plugins/0.log"