Sep 16 00:31:58.902175 ip-10-0-131-194 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 16 00:31:58.902303 ip-10-0-131-194 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 16 00:31:58.902409 ip-10-0-131-194 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 16 00:31:58.902959 ip-10-0-131-194 systemd[1]: Failed to start Kubernetes Kubelet. Sep 16 00:32:09.088273 ip-10-0-131-194 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 16 00:32:09.088288 ip-10-0-131-194 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 097f99fe178144f1a1ae3c5f94ee2517 -- Sep 16 00:34:07.977863 ip-10-0-131-194 systemd[1]: Starting Kubernetes Kubelet... Sep 16 00:34:08.331258 ip-10-0-131-194 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 00:34:08.331258 ip-10-0-131-194 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 00:34:08.331258 ip-10-0-131-194 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 00:34:08.331258 ip-10-0-131-194 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 00:34:08.331258 ip-10-0-131-194 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 00:34:08.332817 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.332735 2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 00:34:08.338809 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338794 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 00:34:08.338809 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338810 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 00:34:08.338876 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338814 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 00:34:08.338876 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338817 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 00:34:08.338876 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338821 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 00:34:08.338876 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338824 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 00:34:08.338876 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338827 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 00:34:08.338876 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338829 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 00:34:08.338876 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338832 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 00:34:08.338876 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338834 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 00:34:08.338876 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338837 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 00:34:08.338876 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338839 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 00:34:08.338876 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338842 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 00:34:08.338876 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338844 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 00:34:08.338876 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338847 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 00:34:08.338876 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338849 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 00:34:08.338876 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338852 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 00:34:08.338876 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338854 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 00:34:08.338876 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338857 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 00:34:08.338876 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338860 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 00:34:08.338876 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338863 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 00:34:08.338876 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338865 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 00:34:08.339345 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338868 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 00:34:08.339345 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338870 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 00:34:08.339345 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338873 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 00:34:08.339345 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338875 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 00:34:08.339345 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338878 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 00:34:08.339345 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338880 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 00:34:08.339345 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338883 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 00:34:08.339345 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338886 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 00:34:08.339345 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338889 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 00:34:08.339345 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338892 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 00:34:08.339345 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338894 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 00:34:08.339345 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338897 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 00:34:08.339345 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338899 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 00:34:08.339345 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338902 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 00:34:08.339345 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338904 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 00:34:08.339345 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338907 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 00:34:08.339345 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338911 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 00:34:08.339345 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338915 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 00:34:08.339345 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338917 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 00:34:08.339827 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338920 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 00:34:08.339827 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338922 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 00:34:08.339827 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338925 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 00:34:08.339827 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338927 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 00:34:08.339827 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338930 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 00:34:08.339827 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338932 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 00:34:08.339827 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338934 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 00:34:08.339827 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338937 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 00:34:08.339827 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338939 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 00:34:08.339827 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338941 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 00:34:08.339827 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338944 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 00:34:08.339827 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338946 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 00:34:08.339827 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338949 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 00:34:08.339827 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338951 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 00:34:08.339827 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338955 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 00:34:08.339827 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338958 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 00:34:08.339827 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338961 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 00:34:08.339827 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338963 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 00:34:08.339827 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338966 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 00:34:08.339827 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338968 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 00:34:08.340315 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338971 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 00:34:08.340315 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338974 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 00:34:08.340315 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338976 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 00:34:08.340315 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338979 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 00:34:08.340315 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338981 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 00:34:08.340315 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338984 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 00:34:08.340315 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338986 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 00:34:08.340315 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338989 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 00:34:08.340315 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338991 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 00:34:08.340315 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338993 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 00:34:08.340315 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.338998 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 00:34:08.340315 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339001 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 00:34:08.340315 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339004 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 00:34:08.340315 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339007 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 00:34:08.340315 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339009 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 00:34:08.340315 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339012 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 00:34:08.340315 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339015 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 00:34:08.340315 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339017 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 00:34:08.340315 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339020 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 00:34:08.340787 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339022 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 00:34:08.340787 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339027 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 00:34:08.340787 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339030 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 00:34:08.340787 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339033 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 00:34:08.340787 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339036 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 00:34:08.340787 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339038 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 00:34:08.340787 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339041 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 16 00:34:08.340787 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339045 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 00:34:08.340787 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339048 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 00:34:08.340787 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339051 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 00:34:08.340787 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339054 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 00:34:08.340787 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339056 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 00:34:08.340787 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339059 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 00:34:08.340787 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339061 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 00:34:08.340787 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339064 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 00:34:08.340787 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339066 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 00:34:08.340787 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339069 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 00:34:08.340787 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339071 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 00:34:08.340787 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339074 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 00:34:08.340787 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339776 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 00:34:08.341306 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339784 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 00:34:08.341306 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339787 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 00:34:08.341306 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339790 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 00:34:08.341306 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339793 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 00:34:08.341306 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339797 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 00:34:08.341306 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339800 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 00:34:08.341306 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339803 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 00:34:08.341306 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339806 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 00:34:08.341306 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339808 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 00:34:08.341306 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339811 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 00:34:08.341306 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339814 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 16 00:34:08.341306 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339816 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 00:34:08.341306 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339819 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 00:34:08.341306 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339822 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 00:34:08.341306 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339824 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 00:34:08.341306 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339827 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 00:34:08.341306 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339829 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 00:34:08.341306 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339832 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 00:34:08.341306 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339834 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 00:34:08.341306 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339837 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 00:34:08.341843 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339840 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 00:34:08.341843 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339843 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 00:34:08.341843 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339845 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 00:34:08.341843 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339848 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 00:34:08.341843 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339851 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 00:34:08.341843 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339853 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 00:34:08.341843 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339855 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 00:34:08.341843 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339858 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 00:34:08.341843 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339860 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 00:34:08.341843 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339863 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 00:34:08.341843 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339866 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 00:34:08.341843 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339869 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 00:34:08.341843 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339871 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 00:34:08.341843 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339875 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 00:34:08.341843 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339879 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 00:34:08.341843 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339882 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 00:34:08.341843 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339885 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 00:34:08.341843 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339889 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 00:34:08.341843 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339892 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 00:34:08.341843 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339895 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 00:34:08.342332 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339897 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 00:34:08.342332 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339900 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 00:34:08.342332 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339903 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 00:34:08.342332 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339906 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 00:34:08.342332 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339908 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 00:34:08.342332 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339912 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 00:34:08.342332 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339916 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 00:34:08.342332 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339919 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 00:34:08.342332 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339922 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 00:34:08.342332 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339925 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 00:34:08.342332 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339927 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 00:34:08.342332 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339930 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 00:34:08.342332 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339933 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 00:34:08.342332 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339935 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 00:34:08.342332 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339938 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 00:34:08.342332 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339941 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 00:34:08.342332 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339943 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 00:34:08.342332 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339945 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 00:34:08.342332 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339949 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339953 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339955 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339958 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339960 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339963 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339965 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339968 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339971 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339973 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339976 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339978 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339982 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339985 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339987 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339990 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339993 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339995 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.339998 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340000 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340003 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 00:34:08.342818 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340005 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 00:34:08.343322 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340008 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 00:34:08.343322 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340010 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 00:34:08.343322 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340013 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 00:34:08.343322 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340015 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 00:34:08.343322 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340018 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 00:34:08.343322 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340020 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 00:34:08.343322 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340023 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 00:34:08.343322 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340025 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 00:34:08.343322 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340028 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 00:34:08.343322 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340030 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 00:34:08.343322 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340033 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 00:34:08.343322 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340035 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 00:34:08.343322 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340038 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 00:34:08.343322 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340041 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 00:34:08.343322 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340043 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 00:34:08.343322 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340045 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 00:34:08.343322 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340048 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 00:34:08.343322 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340050 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 00:34:08.343322 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340396 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 00:34:08.343793 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340401 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 00:34:08.343793 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340404 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 00:34:08.343793 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340408 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 00:34:08.343793 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340411 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 00:34:08.343793 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340414 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 00:34:08.343793 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340417 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 00:34:08.343793 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340420 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 00:34:08.343793 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340423 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 00:34:08.343793 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340426 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 00:34:08.343793 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340428 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 00:34:08.343793 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340431 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 00:34:08.343793 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340434 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 00:34:08.343793 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340436 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 00:34:08.343793 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340439 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 00:34:08.343793 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340442 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 00:34:08.343793 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340446 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 00:34:08.343793 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340449 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 00:34:08.343793 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340452 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 00:34:08.343793 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340455 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 00:34:08.343793 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340457 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 00:34:08.344280 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340460 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 00:34:08.344280 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340462 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 00:34:08.344280 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340465 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 00:34:08.344280 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340467 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 00:34:08.344280 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340470 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 00:34:08.344280 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340473 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 00:34:08.344280 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340475 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 00:34:08.344280 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340477 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 00:34:08.344280 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340480 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 00:34:08.344280 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340482 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 00:34:08.344280 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340485 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 00:34:08.344280 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340487 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 00:34:08.344280 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340490 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 00:34:08.344280 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340495 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 00:34:08.344280 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340497 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 00:34:08.344280 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340500 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 00:34:08.344280 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340508 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 00:34:08.344280 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340511 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 00:34:08.344280 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340514 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 00:34:08.344765 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340517 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 00:34:08.344765 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340519 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 00:34:08.344765 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340522 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 00:34:08.344765 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340524 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 00:34:08.344765 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340527 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 00:34:08.344765 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340529 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 00:34:08.344765 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340532 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 00:34:08.344765 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340535 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 00:34:08.344765 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340539 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 00:34:08.344765 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340543 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 16 00:34:08.344765 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340546 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 00:34:08.344765 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340549 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 00:34:08.344765 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340551 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 00:34:08.344765 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340554 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 00:34:08.344765 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340557 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 00:34:08.344765 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340559 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 00:34:08.344765 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340562 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 00:34:08.344765 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340565 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 00:34:08.344765 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340568 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 00:34:08.344765 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340570 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 00:34:08.345226 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340588 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 00:34:08.345226 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340591 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 00:34:08.345226 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340594 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 00:34:08.345226 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340596 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 00:34:08.345226 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340599 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 00:34:08.345226 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340601 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 00:34:08.345226 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340607 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 00:34:08.345226 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340610 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 00:34:08.345226 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340613 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 00:34:08.345226 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340616 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 00:34:08.345226 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340619 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 00:34:08.345226 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340623 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 00:34:08.345226 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340626 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 00:34:08.345226 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340628 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 00:34:08.345226 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340631 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 00:34:08.345226 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340633 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 00:34:08.345226 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340636 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 00:34:08.345226 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340639 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 00:34:08.345226 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340641 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 00:34:08.345717 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340644 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 00:34:08.345717 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340646 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 00:34:08.345717 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340649 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 00:34:08.345717 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340652 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 00:34:08.345717 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340654 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 00:34:08.345717 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340657 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 00:34:08.345717 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340659 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 00:34:08.345717 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340661 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 00:34:08.345717 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340664 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 00:34:08.345717 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340666 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 00:34:08.345717 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340669 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 00:34:08.345717 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340671 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 00:34:08.345717 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340674 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 00:34:08.345717 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340676 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 00:34:08.345717 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340679 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 00:34:08.345717 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340681 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 00:34:08.345717 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340683 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 00:34:08.345717 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340686 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 00:34:08.345717 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340688 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.340692 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340769 2567 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340802 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340808 2567 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340812 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340818 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340822 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340826 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340830 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340833 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340836 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340840 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340843 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340846 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340849 2567 flags.go:64] FLAG: --cgroup-root="" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340852 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340854 2567 flags.go:64] FLAG: --client-ca-file="" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340857 2567 flags.go:64] FLAG: --cloud-provider="external" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340860 2567 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340864 2567 flags.go:64] FLAG: --cluster-domain="" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340867 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340870 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340873 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 00:34:08.346224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340876 2567 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340880 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340883 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340886 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340889 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340892 2567 flags.go:64] FLAG: --contention-profiling="false" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340895 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340897 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340900 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340904 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340908 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340911 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340913 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340918 2567 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340921 2567 flags.go:64] FLAG: --enable-server="true" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340924 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340928 2567 flags.go:64] FLAG: --event-burst="100" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340931 2567 flags.go:64] FLAG: --event-qps="50" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340934 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340937 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340940 2567 flags.go:64] FLAG: --eviction-hard="" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340943 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340946 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340949 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340952 2567 flags.go:64] FLAG: --eviction-soft="" Sep 16 00:34:08.346828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340955 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340958 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340961 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340964 2567 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340967 2567 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340970 2567 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340972 2567 flags.go:64] FLAG: --feature-gates="" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340976 2567 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340979 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340982 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340985 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340988 2567 flags.go:64] FLAG: --healthz-port="10248" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340991 2567 flags.go:64] FLAG: --help="false" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340993 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-131-194.ec2.internal" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340996 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.340999 2567 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341002 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341006 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341009 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341012 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341016 2567 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341019 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341021 2567 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 00:34:08.347418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341025 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341028 2567 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341031 2567 flags.go:64] FLAG: --kube-reserved="" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341034 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341036 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341039 2567 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341042 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341045 2567 flags.go:64] FLAG: --lock-file="" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341047 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341050 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341052 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341057 2567 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341060 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341063 2567 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341065 2567 flags.go:64] FLAG: --logging-format="text" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341068 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341071 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341074 2567 flags.go:64] FLAG: --manifest-url="" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341077 2567 flags.go:64] FLAG: --manifest-url-header="" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341081 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341084 2567 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341088 2567 flags.go:64] FLAG: --max-pods="110" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341091 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341093 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341096 2567 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 00:34:08.348022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341099 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341102 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341105 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341108 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341115 2567 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341118 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341121 2567 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341124 2567 flags.go:64] FLAG: --pod-cidr="" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341127 2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341133 2567 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341136 2567 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341139 2567 flags.go:64] FLAG: --pods-per-core="0" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341142 2567 flags.go:64] FLAG: --port="10250" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341145 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341147 2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e2189ab1908390ef" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341151 2567 flags.go:64] FLAG: --qos-reserved="" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341154 2567 flags.go:64] FLAG: --read-only-port="10255" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341157 2567 flags.go:64] FLAG: --register-node="true" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341160 2567 flags.go:64] FLAG: --register-with-taints="" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341163 2567 flags.go:64] FLAG: --registry-burst="10" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341166 2567 flags.go:64] FLAG: --registry-qps="5" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341168 2567 flags.go:64] FLAG: --reserved-cpus="" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341171 2567 flags.go:64] FLAG: --reserved-memory="" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341175 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341178 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 00:34:08.348643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341181 2567 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341183 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341186 2567 flags.go:64] FLAG: --runonce="false" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341189 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341192 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341195 2567 flags.go:64] FLAG: --seccomp-default="false" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341197 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341200 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341203 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341206 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341209 2567 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341212 2567 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341216 2567 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341219 2567 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341222 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341225 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341228 2567 flags.go:64] FLAG: --system-cgroups="" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341230 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341235 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341238 2567 flags.go:64] FLAG: --tls-cert-file="" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341241 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341244 2567 flags.go:64] FLAG: --tls-min-version="" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341247 2567 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341250 2567 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341253 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 00:34:08.349337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341256 2567 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 00:34:08.350003 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341258 2567 flags.go:64] FLAG: --v="2" Sep 16 00:34:08.350003 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341262 2567 flags.go:64] FLAG: --version="false" Sep 16 00:34:08.350003 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341266 2567 flags.go:64] FLAG: --vmodule="" Sep 16 00:34:08.350003 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341269 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 00:34:08.350003 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341272 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 00:34:08.350003 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341378 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 00:34:08.350003 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341382 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 00:34:08.350003 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341385 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 00:34:08.350003 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341389 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 00:34:08.350003 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341392 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 00:34:08.350003 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341395 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 00:34:08.350003 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341399 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 00:34:08.350003 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341402 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 00:34:08.350003 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341405 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 00:34:08.350003 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341409 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 00:34:08.350003 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341413 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 00:34:08.350003 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341415 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 00:34:08.350003 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341418 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 00:34:08.350003 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341422 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 00:34:08.350003 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341426 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 00:34:08.350501 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341429 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 00:34:08.350501 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341432 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 00:34:08.350501 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341435 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 00:34:08.350501 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341438 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 16 00:34:08.350501 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341441 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 00:34:08.350501 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341444 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 00:34:08.350501 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341447 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 00:34:08.350501 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341450 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 00:34:08.350501 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341452 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 00:34:08.350501 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341455 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 00:34:08.350501 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341458 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 00:34:08.350501 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341460 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 00:34:08.350501 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341463 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 00:34:08.350501 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341465 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 00:34:08.350501 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341467 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 00:34:08.350501 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341470 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 00:34:08.350501 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341472 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 00:34:08.350501 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341475 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 00:34:08.350501 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341477 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 00:34:08.350501 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341480 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 00:34:08.351005 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341483 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 00:34:08.351005 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341485 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 00:34:08.351005 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341488 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 00:34:08.351005 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341491 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 00:34:08.351005 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341494 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 00:34:08.351005 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341497 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 00:34:08.351005 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341499 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 00:34:08.351005 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341502 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 00:34:08.351005 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341504 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 00:34:08.351005 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341507 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 00:34:08.351005 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341510 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 00:34:08.351005 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341513 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 00:34:08.351005 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341516 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 00:34:08.351005 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341518 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 00:34:08.351005 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341521 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 00:34:08.351005 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341523 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 00:34:08.351005 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341526 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 00:34:08.351005 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341529 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 00:34:08.351005 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341531 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 00:34:08.351465 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341534 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 00:34:08.351465 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341536 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 00:34:08.351465 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341539 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 00:34:08.351465 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341541 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 00:34:08.351465 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341544 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 00:34:08.351465 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341547 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 00:34:08.351465 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341549 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 00:34:08.351465 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341552 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 00:34:08.351465 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341554 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 00:34:08.351465 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341557 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 00:34:08.351465 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341559 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 00:34:08.351465 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341562 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 00:34:08.351465 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341565 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 00:34:08.351465 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341567 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 00:34:08.351465 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341570 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 00:34:08.351465 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341572 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 00:34:08.351465 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341591 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 00:34:08.351465 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341594 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 00:34:08.351465 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341596 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 00:34:08.351465 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341599 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 00:34:08.352042 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341609 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 00:34:08.352042 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341613 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 00:34:08.352042 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341615 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 00:34:08.352042 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341619 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 00:34:08.352042 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341622 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 00:34:08.352042 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341625 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 00:34:08.352042 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341627 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 00:34:08.352042 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341630 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 00:34:08.352042 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341632 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 00:34:08.352042 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341635 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 00:34:08.352042 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341639 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 00:34:08.352042 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341641 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 00:34:08.352042 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341644 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 00:34:08.352042 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341647 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 00:34:08.352042 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341649 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 00:34:08.352042 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341652 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 00:34:08.352042 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341654 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 00:34:08.352042 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341657 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 00:34:08.352042 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341659 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 00:34:08.352042 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341662 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 00:34:08.352525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341665 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 00:34:08.352525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341667 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 00:34:08.352525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341670 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 00:34:08.352525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341672 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 00:34:08.352525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.341676 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 00:34:08.352525 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.341685 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 00:34:08.352525 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.350428 2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 00:34:08.352525 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.350442 2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 00:34:08.352525 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.350457 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 00:34:08.352525 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.350463 2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 00:34:08.352525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350517 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 00:34:08.352525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350522 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 00:34:08.352525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350526 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 00:34:08.352525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350529 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 00:34:08.352929 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350531 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 00:34:08.352929 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350534 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 00:34:08.352929 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350537 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 00:34:08.352929 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350539 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 00:34:08.352929 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350542 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 00:34:08.352929 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350545 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 00:34:08.352929 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350548 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 00:34:08.352929 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350550 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 00:34:08.352929 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350554 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 00:34:08.352929 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350557 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 00:34:08.352929 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350559 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 00:34:08.352929 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350562 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 00:34:08.352929 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350565 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 00:34:08.352929 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350567 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 00:34:08.352929 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350570 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 00:34:08.352929 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350572 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 00:34:08.352929 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350590 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 00:34:08.352929 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350593 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 00:34:08.352929 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350596 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 00:34:08.352929 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350600 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 00:34:08.353402 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350602 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 00:34:08.353402 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350605 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 00:34:08.353402 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350608 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 00:34:08.353402 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350610 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 00:34:08.353402 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350613 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 00:34:08.353402 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350616 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 00:34:08.353402 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350619 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 00:34:08.353402 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350622 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 00:34:08.353402 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350625 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 00:34:08.353402 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350627 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 00:34:08.353402 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350630 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 00:34:08.353402 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350633 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 00:34:08.353402 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350636 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 00:34:08.353402 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350638 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 00:34:08.353402 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350641 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 00:34:08.353402 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350644 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 16 00:34:08.353402 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350646 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 00:34:08.353402 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350649 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 00:34:08.353402 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350651 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 00:34:08.353402 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350654 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 00:34:08.353952 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350656 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 00:34:08.353952 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350660 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 00:34:08.353952 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350662 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 00:34:08.353952 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350665 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 00:34:08.353952 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350668 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 00:34:08.353952 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350670 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 00:34:08.353952 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350672 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 00:34:08.353952 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350675 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 00:34:08.353952 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350677 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 00:34:08.353952 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350681 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 00:34:08.353952 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350684 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 00:34:08.353952 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350686 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 00:34:08.353952 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350690 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 00:34:08.353952 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350694 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 00:34:08.353952 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350697 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 00:34:08.353952 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350700 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 00:34:08.353952 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350703 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 00:34:08.353952 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350706 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 00:34:08.353952 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350709 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 00:34:08.354408 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350712 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 00:34:08.354408 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350715 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 00:34:08.354408 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350717 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 00:34:08.354408 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350720 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 00:34:08.354408 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350723 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 00:34:08.354408 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350726 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 00:34:08.354408 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350728 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 00:34:08.354408 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350731 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 00:34:08.354408 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350733 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 00:34:08.354408 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350736 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 00:34:08.354408 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350739 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 00:34:08.354408 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350741 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 00:34:08.354408 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350744 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 00:34:08.354408 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350746 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 00:34:08.354408 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350748 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 00:34:08.354408 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350751 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 00:34:08.354408 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350754 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 00:34:08.354408 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350756 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 00:34:08.354408 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350759 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 00:34:08.354408 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350761 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 00:34:08.354923 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350764 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 00:34:08.354923 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350766 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 00:34:08.354923 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350769 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 00:34:08.354923 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350772 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 00:34:08.354923 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350774 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 00:34:08.354923 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350777 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 00:34:08.354923 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350779 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 00:34:08.354923 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350781 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 00:34:08.354923 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350784 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 00:34:08.354923 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350786 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 00:34:08.354923 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350790 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 00:34:08.354923 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350794 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 00:34:08.354923 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350797 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 00:34:08.354923 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350799 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 00:34:08.354923 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350803 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 00:34:08.354923 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350807 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 00:34:08.355312 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.350812 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 00:34:08.355312 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350917 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 00:34:08.355312 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350922 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 00:34:08.355312 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350925 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 00:34:08.355312 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350928 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 00:34:08.355312 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350932 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 00:34:08.355312 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350935 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 00:34:08.355312 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350938 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 00:34:08.355312 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350941 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 00:34:08.355312 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350944 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 00:34:08.355312 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350947 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 00:34:08.355312 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350950 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 00:34:08.355312 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350954 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 00:34:08.355312 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350957 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 00:34:08.355686 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350959 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 00:34:08.355686 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350962 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 00:34:08.355686 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350964 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 00:34:08.355686 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350967 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 00:34:08.355686 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350969 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 00:34:08.355686 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350972 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 00:34:08.355686 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350974 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 00:34:08.355686 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350977 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 00:34:08.355686 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350979 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 00:34:08.355686 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350982 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 00:34:08.355686 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350984 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 00:34:08.355686 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350987 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 00:34:08.355686 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350989 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 00:34:08.355686 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350992 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 00:34:08.355686 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350995 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 00:34:08.355686 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.350998 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 00:34:08.355686 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351000 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 00:34:08.355686 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351003 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 00:34:08.355686 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351005 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 00:34:08.356128 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351008 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 00:34:08.356128 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351011 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 00:34:08.356128 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351014 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 00:34:08.356128 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351017 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 00:34:08.356128 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351020 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 00:34:08.356128 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351024 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 00:34:08.356128 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351027 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 00:34:08.356128 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351030 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 00:34:08.356128 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351033 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 00:34:08.356128 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351036 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 00:34:08.356128 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351038 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 00:34:08.356128 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351041 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 00:34:08.356128 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351044 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 00:34:08.356128 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351046 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 00:34:08.356128 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351049 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 00:34:08.356128 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351052 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 00:34:08.356128 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351054 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 00:34:08.356128 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351057 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 00:34:08.356128 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351059 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 00:34:08.356128 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351062 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 00:34:08.356726 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351064 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 00:34:08.356726 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351067 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 00:34:08.356726 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351069 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 00:34:08.356726 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351072 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 00:34:08.356726 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351075 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 00:34:08.356726 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351077 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 00:34:08.356726 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351080 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 00:34:08.356726 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351082 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 00:34:08.356726 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351085 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 16 00:34:08.356726 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351087 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 00:34:08.356726 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351090 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 00:34:08.356726 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351092 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 00:34:08.356726 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351095 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 00:34:08.356726 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351097 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 00:34:08.356726 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351100 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 00:34:08.356726 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351103 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 00:34:08.356726 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351105 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 00:34:08.356726 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351108 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 00:34:08.356726 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351110 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 00:34:08.356726 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351113 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 00:34:08.357525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351115 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 00:34:08.357525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351118 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 00:34:08.357525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351120 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 00:34:08.357525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351123 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 00:34:08.357525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351125 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 00:34:08.357525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351129 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 00:34:08.357525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351131 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 00:34:08.357525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351135 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 00:34:08.357525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351138 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 00:34:08.357525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351141 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 00:34:08.357525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351144 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 00:34:08.357525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351146 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 00:34:08.357525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351149 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 00:34:08.357525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351152 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 00:34:08.357525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351154 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 00:34:08.357525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351157 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 00:34:08.357525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351159 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 00:34:08.357525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351162 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 00:34:08.357525 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351164 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 00:34:08.358067 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351167 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 00:34:08.358067 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351169 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 00:34:08.358067 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351172 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 00:34:08.358067 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351174 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 00:34:08.358067 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351177 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 00:34:08.358067 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351179 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 00:34:08.358067 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351182 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 00:34:08.358067 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:08.351185 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 00:34:08.358067 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.351189 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 00:34:08.358067 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.351282 2567 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 00:34:08.358067 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.353504 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 00:34:08.358067 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.354380 2567 server.go:1019] "Starting client certificate rotation" Sep 16 00:34:08.358067 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.354480 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 00:34:08.358067 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.355181 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 00:34:08.358970 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.358948 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 00:34:08.360628 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.360601 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 00:34:08.373071 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.373053 2567 log.go:25] "Validated CRI v1 runtime API" Sep 16 00:34:08.376793 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.376775 2567 log.go:25] "Validated CRI v1 image API" Sep 16 00:34:08.377764 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.377738 2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 00:34:08.380917 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.380886 2567 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 eda03f60-c6ad-43dc-8455-ba85a222aaeb:/dev/nvme0n1p4 fbdd3e0f-7436-4a47-81f5-c2a60751d693:/dev/nvme0n1p3] Sep 16 00:34:08.380963 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.380911 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 00:34:08.385711 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.385599 2567 manager.go:216] Machine: {Timestamp:2026-09-16 00:34:08.384413868 +0000 UTC m=+0.298096307 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100057 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b89af8f942f6ddb906a569500c334 SystemUUID:ec2b89af-8f94-2f6d-db90-6a569500c334 BootID:097f99fe-1781-44f1-a1ae-3c5f94ee2517 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f2:1d:bc:61 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f2:1d:bc:61 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:b2:07:bb:33:2c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 00:34:08.385711 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.385706 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 00:34:08.385827 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.385779 2567 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 00:34:08.389587 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.389548 2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 00:34:08.389727 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.389590 2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-194.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 00:34:08.391131 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.391120 2567 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 00:34:08.391171 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.391134 2567 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 00:34:08.391171 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.391142 2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 00:34:08.391878 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.391862 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 00:34:08.392948 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.392938 2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 00:34:08.395181 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.395162 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 00:34:08.397038 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.397026 2567 state_mem.go:36] "Initialized new in-memory state store" Sep 16 00:34:08.397142 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.397133 2567 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 00:34:08.397172 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.397162 2567 kubelet.go:486] "Attempting to sync node with API server" Sep 16 00:34:08.397199 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.397177 2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 00:34:08.397199 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.397190 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 00:34:08.397199 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.397198 2567 kubelet.go:398] "Adding apiserver pod source" Sep 16 00:34:08.397300 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.397207 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 00:34:08.397627 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.397612 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 00:34:08.397665 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.397645 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 00:34:08.400615 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.400596 2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 00:34:08.403278 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.403264 2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 00:34:08.403337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.403295 2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 00:34:08.405328 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.405317 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 00:34:08.405394 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.405332 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 00:34:08.405394 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.405339 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 00:34:08.405394 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.405344 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 00:34:08.405394 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.405352 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 00:34:08.405394 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.405358 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 00:34:08.405394 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.405370 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 00:34:08.405394 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.405375 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 00:34:08.405394 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.405382 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 00:34:08.405394 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.405387 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 00:34:08.405394 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.405395 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 00:34:08.405674 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.405400 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 00:34:08.406218 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.406208 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 00:34:08.406218 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.406218 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 00:34:08.407856 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:08.407834 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 00:34:08.407856 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:08.407838 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 00:34:08.409801 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.409789 2567 server.go:1268] "Started kubelet" Sep 16 00:34:08.409902 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.409877 2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 00:34:08.410025 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.409869 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 00:34:08.410095 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.410048 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 00:34:08.410194 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.410174 2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-194.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 00:34:08.410525 ip-10-0-131-194 systemd[1]: Started Kubernetes Kubelet. Sep 16 00:34:08.411003 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.410986 2567 server.go:310] "Adding debug handlers to kubelet server" Sep 16 00:34:08.411546 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.411527 2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 00:34:08.414633 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:08.414617 2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 00:34:08.417851 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.417834 2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 00:34:08.417851 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.417842 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 00:34:08.417997 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.417836 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kdqlx" Sep 16 00:34:08.417997 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:08.417960 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-194.ec2.internal\" not found" Sep 16 00:34:08.418484 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.418364 2567 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 00:34:08.418484 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.418389 2567 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 00:34:08.418484 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.418374 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 00:34:08.418484 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.418465 2567 reconstruct.go:97] "Volume reconstruction finished" Sep 16 00:34:08.418484 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.418471 2567 reconciler.go:29] "Reconciler: start to sync state" Sep 16 00:34:08.418764 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.418720 2567 factory.go:55] Registering systemd factory Sep 16 00:34:08.418764 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.418754 2567 factory.go:223] Registration of the systemd container factory successfully Sep 16 00:34:08.418941 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.418928 2567 factory.go:153] Registering CRI-O factory Sep 16 00:34:08.418993 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.418947 2567 factory.go:223] Registration of the crio container factory successfully Sep 16 00:34:08.419031 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.418997 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 00:34:08.419031 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.419015 2567 factory.go:103] Registering Raw factory Sep 16 00:34:08.419031 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.419027 2567 manager.go:1196] Started watching for new ooms in manager Sep 16 00:34:08.419186 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:08.418223 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-194.ec2.internal.18d5a5e94e547ba9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-194.ec2.internal,UID:ip-10-0-131-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-194.ec2.internal,},FirstTimestamp:2026-09-16 00:34:08.409762729 +0000 UTC m=+0.323445155,LastTimestamp:2026-09-16 00:34:08.409762729 +0000 UTC m=+0.323445155,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-194.ec2.internal,}" Sep 16 00:34:08.419415 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.419402 2567 manager.go:349] Starting recovery of all containers Sep 16 00:34:08.423299 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:08.423254 2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 00:34:08.423411 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:08.423374 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 00:34:08.429454 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.429406 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kdqlx" Sep 16 00:34:08.431496 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.431480 2567 manager.go:354] Recovery completed Sep 16 00:34:08.435498 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.435487 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 00:34:08.438002 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.437984 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasSufficientMemory" Sep 16 00:34:08.438062 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.438011 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 00:34:08.438062 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.438026 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasSufficientPID" Sep 16 00:34:08.438617 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.438604 2567 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 00:34:08.438617 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.438616 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 00:34:08.438731 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.438618 2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 00:34:08.438731 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.438656 2567 state_mem.go:36] "Initialized new in-memory state store" Sep 16 00:34:08.439488 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.439465 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasSufficientMemory" Sep 16 00:34:08.439558 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.439499 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 00:34:08.439558 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.439514 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasSufficientPID" Sep 16 00:34:08.440156 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:08.440093 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-194.ec2.internal.18d5a5e950035992 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-194.ec2.internal,UID:ip-10-0-131-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-194.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-194.ec2.internal,},FirstTimestamp:2026-09-16 00:34:08.438000018 +0000 UTC m=+0.351682443,LastTimestamp:2026-09-16 00:34:08.438000018 +0000 UTC m=+0.351682443,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-194.ec2.internal,}" Sep 16 00:34:08.440687 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.440671 2567 policy_none.go:49] "None policy: Start" Sep 16 00:34:08.440759 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.440691 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 00:34:08.440759 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.440704 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 00:34:08.442695 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.442681 2567 policy_none.go:47] "Start" Sep 16 00:34:08.478223 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.478208 2567 manager.go:337] "Starting Device Plugin manager" Sep 16 00:34:08.478299 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:08.478250 2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 00:34:08.478299 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.478263 2567 server.go:87] "Starting device plugin registration server" Sep 16 00:34:08.478500 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.478486 2567 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 00:34:08.478549 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.478501 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 00:34:08.478620 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.478598 2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 00:34:08.478694 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.478668 2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 00:34:08.478694 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.478681 2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 00:34:08.479186 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:08.479166 2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 00:34:08.479259 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:08.479221 2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-194.ec2.internal\" not found" Sep 16 00:34:08.559249 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.559226 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 00:34:08.560876 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.560543 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 00:34:08.560876 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.560599 2567 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 00:34:08.560876 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.560627 2567 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 00:34:08.560876 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:08.560669 2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 00:34:08.562719 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.562700 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 00:34:08.579631 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.579606 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 00:34:08.580679 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.580666 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasSufficientMemory" Sep 16 00:34:08.580743 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.580690 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 00:34:08.580743 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.580699 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasSufficientPID" Sep 16 00:34:08.580743 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.580718 2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-194.ec2.internal" Sep 16 00:34:08.588839 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.588803 2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-194.ec2.internal" Sep 16 00:34:08.588839 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:08.588822 2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-194.ec2.internal\": node \"ip-10-0-131-194.ec2.internal\" not found" Sep 16 00:34:08.602431 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:08.602413 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-194.ec2.internal\" not found" Sep 16 00:34:08.660902 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.660879 2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-194.ec2.internal"] Sep 16 00:34:08.660969 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.660949 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 00:34:08.662688 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.662671 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasSufficientMemory" Sep 16 00:34:08.662786 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.662699 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 00:34:08.662786 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.662709 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasSufficientPID" Sep 16 00:34:08.664474 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.664462 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 00:34:08.664620 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.664606 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal" Sep 16 00:34:08.664670 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.664634 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 00:34:08.665114 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.665099 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasSufficientMemory" Sep 16 00:34:08.665199 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.665121 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasSufficientMemory" Sep 16 00:34:08.665199 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.665128 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 00:34:08.665199 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.665143 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 00:34:08.665199 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.665147 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasSufficientPID" Sep 16 00:34:08.665199 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.665153 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasSufficientPID" Sep 16 00:34:08.666973 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.666957 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-194.ec2.internal" Sep 16 00:34:08.667056 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.666985 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 00:34:08.667609 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.667595 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasSufficientMemory" Sep 16 00:34:08.667693 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.667622 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 00:34:08.667693 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.667637 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeHasSufficientPID" Sep 16 00:34:08.691145 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:08.691125 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-194.ec2.internal\" not found" node="ip-10-0-131-194.ec2.internal" Sep 16 00:34:08.695471 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:08.695457 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-194.ec2.internal\" not found" node="ip-10-0-131-194.ec2.internal" Sep 16 00:34:08.703139 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:08.703124 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-194.ec2.internal\" not found" Sep 16 00:34:08.720565 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.720547 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa7d14c1ae359f30b74b1f397edcabc9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal\" (UID: \"fa7d14c1ae359f30b74b1f397edcabc9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal" Sep 16 00:34:08.720638 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.720571 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa7d14c1ae359f30b74b1f397edcabc9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal\" (UID: \"fa7d14c1ae359f30b74b1f397edcabc9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal" Sep 16 00:34:08.720638 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.720619 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67a2beadf904429fdc039f04a692727f-config\") pod \"kube-apiserver-proxy-ip-10-0-131-194.ec2.internal\" (UID: \"67a2beadf904429fdc039f04a692727f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-194.ec2.internal" Sep 16 00:34:08.804114 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:08.804084 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-194.ec2.internal\" not found" Sep 16 00:34:08.821510 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.821489 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa7d14c1ae359f30b74b1f397edcabc9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal\" (UID: \"fa7d14c1ae359f30b74b1f397edcabc9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal" Sep 16 00:34:08.821569 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.821514 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa7d14c1ae359f30b74b1f397edcabc9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal\" (UID: \"fa7d14c1ae359f30b74b1f397edcabc9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal" Sep 16 00:34:08.821569 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.821530 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67a2beadf904429fdc039f04a692727f-config\") pod \"kube-apiserver-proxy-ip-10-0-131-194.ec2.internal\" (UID: \"67a2beadf904429fdc039f04a692727f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-194.ec2.internal" Sep 16 00:34:08.821569 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.821557 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67a2beadf904429fdc039f04a692727f-config\") pod \"kube-apiserver-proxy-ip-10-0-131-194.ec2.internal\" (UID: \"67a2beadf904429fdc039f04a692727f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-194.ec2.internal" Sep 16 00:34:08.821696 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.821590 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa7d14c1ae359f30b74b1f397edcabc9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal\" (UID: \"fa7d14c1ae359f30b74b1f397edcabc9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal" Sep 16 00:34:08.821696 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.821614 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa7d14c1ae359f30b74b1f397edcabc9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal\" (UID: \"fa7d14c1ae359f30b74b1f397edcabc9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal" Sep 16 00:34:08.904362 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:08.904309 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-194.ec2.internal\" not found" Sep 16 00:34:08.994802 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.994772 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal" Sep 16 00:34:08.999670 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:08.999554 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-194.ec2.internal" Sep 16 00:34:09.004482 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:09.004356 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-194.ec2.internal\" not found" Sep 16 00:34:09.104841 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:09.104809 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-194.ec2.internal\" not found" Sep 16 00:34:09.205324 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:09.205261 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-194.ec2.internal\" not found" Sep 16 00:34:09.305829 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:09.305799 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-194.ec2.internal\" not found" Sep 16 00:34:09.332815 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:09.332793 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 00:34:09.354413 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:09.354390 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 00:34:09.354516 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:09.354501 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 00:34:09.354561 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:09.354535 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 00:34:09.405865 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:09.405849 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-194.ec2.internal\" not found" Sep 16 00:34:09.433843 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:09.433808 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 00:29:09 +0000 UTC" deadline="2028-03-17 00:03:03.160481228 +0000 UTC" Sep 16 00:34:09.433843 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:09.433838 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13151h28m53.726645752s" Sep 16 00:34:09.506146 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:09.506097 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-194.ec2.internal\" not found" Sep 16 00:34:09.553086 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:09.553054 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa7d14c1ae359f30b74b1f397edcabc9.slice/crio-839fb940a0de3a2b12b6018e330d1d057751bff77f633dc699b4d842fa5a7ddb WatchSource:0}: Error finding container 839fb940a0de3a2b12b6018e330d1d057751bff77f633dc699b4d842fa5a7ddb: Status 404 returned error can't find the container with id 839fb940a0de3a2b12b6018e330d1d057751bff77f633dc699b4d842fa5a7ddb Sep 16 00:34:09.553332 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:09.553315 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a2beadf904429fdc039f04a692727f.slice/crio-4021ee132ce6d82497bca308400427e02d9473b979be45170d4a7de55451934f WatchSource:0}: Error finding container 4021ee132ce6d82497bca308400427e02d9473b979be45170d4a7de55451934f: Status 404 returned error can't find the container with id 4021ee132ce6d82497bca308400427e02d9473b979be45170d4a7de55451934f Sep 16 00:34:09.557477 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:09.557462 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 00:34:09.566505 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:09.566465 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-194.ec2.internal" event={"ID":"67a2beadf904429fdc039f04a692727f","Type":"ContainerStarted","Data":"4021ee132ce6d82497bca308400427e02d9473b979be45170d4a7de55451934f"} Sep 16 00:34:09.567405 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:09.567385 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal" event={"ID":"fa7d14c1ae359f30b74b1f397edcabc9","Type":"ContainerStarted","Data":"839fb940a0de3a2b12b6018e330d1d057751bff77f633dc699b4d842fa5a7ddb"} Sep 16 00:34:09.586226 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:09.586204 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 00:34:09.607214 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:09.607193 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-194.ec2.internal\" not found" Sep 16 00:34:09.707728 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:09.707705 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-194.ec2.internal\" not found" Sep 16 00:34:09.808202 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:09.808152 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-194.ec2.internal\" not found" Sep 16 00:34:09.908981 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:09.908959 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-194.ec2.internal\" not found" Sep 16 00:34:09.990331 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:09.990305 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 00:34:10.018739 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:10.018717 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal" Sep 16 00:34:10.031302 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:10.031284 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 00:34:10.032399 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:10.032225 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-194.ec2.internal" Sep 16 00:34:10.040483 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:10.040464 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 00:34:11.949276 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.949205 2567 apiserver.go:52] "Watching apiserver" Sep 16 00:34:11.958376 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.958343 2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 00:34:11.960336 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.960298 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-75ccc","kube-system/global-pull-secret-syncer-fpm24","kube-system/konnectivity-agent-jzl25","kube-system/kube-apiserver-proxy-ip-10-0-131-194.ec2.internal","openshift-cluster-node-tuning-operator/tuned-26586","openshift-dns/node-resolver-cd4mj","openshift-image-registry/node-ca-q86vf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal","openshift-multus/network-metrics-daemon-rxnhx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb","openshift-multus/multus-additional-cni-plugins-9spsl","openshift-multus/multus-bglnh","openshift-network-diagnostics/network-check-target-gw589","openshift-network-operator/iptables-alerter-cb4gp"] Sep 16 00:34:11.962953 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.962932 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:11.965145 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.965123 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:11.965253 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:11.965197 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:11.965564 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.965541 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 00:34:11.965853 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.965833 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 00:34:11.965941 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.965858 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fdlmx\"" Sep 16 00:34:11.966212 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.966100 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 00:34:11.967378 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.967356 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jzl25" Sep 16 00:34:11.970257 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.969800 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cb4gp" Sep 16 00:34:11.970257 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.969817 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 00:34:11.970257 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.969962 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 00:34:11.970257 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.970188 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 00:34:11.970257 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.970194 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c8hlj\"" Sep 16 00:34:11.971763 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.971733 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:11.972592 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.972182 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 00:34:11.972592 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.972344 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vl49d\"" Sep 16 00:34:11.972592 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.972489 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 00:34:11.972592 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.972522 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 00:34:11.974076 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.974056 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v7zh2\"" Sep 16 00:34:11.974076 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.974070 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 00:34:11.974211 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.974059 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 00:34:11.974211 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.974096 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cd4mj" Sep 16 00:34:11.976449 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.976430 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 00:34:11.976539 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.976474 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ndqdx\"" Sep 16 00:34:11.976597 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.976582 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 00:34:11.976777 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.976759 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:11.977165 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:11.976818 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:11.979654 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.979618 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q86vf" Sep 16 00:34:11.981618 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.981599 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:11.981700 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:11.981676 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:11.982219 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.982186 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 00:34:11.982337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.982317 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8tv9d\"" Sep 16 00:34:11.982449 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.982425 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 00:34:11.982516 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.982432 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 00:34:11.982555 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.982517 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 00:34:11.984050 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984021 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:11.984458 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984052 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-systemd\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:11.984458 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984190 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca719833-aa0c-4fe5-8aa4-019e479b8e10-sys-fs\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:11.984458 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984220 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-sysctl-d\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:11.984458 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984243 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-tuned\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:11.984458 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984267 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca719833-aa0c-4fe5-8aa4-019e479b8e10-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:11.984458 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984291 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca719833-aa0c-4fe5-8aa4-019e479b8e10-device-dir\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:11.984458 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984315 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1cd8f1ba-a180-412d-ac9b-8853de8af7ae-agent-certs\") pod \"konnectivity-agent-jzl25\" (UID: \"1cd8f1ba-a180-412d-ac9b-8853de8af7ae\") " pod="kube-system/konnectivity-agent-jzl25" Sep 16 00:34:11.984458 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984337 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-sysconfig\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:11.984458 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984372 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-lib-modules\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:11.984458 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984407 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63a33f01-342a-4908-a744-795debfc5de4-host-slash\") pod \"iptables-alerter-cb4gp\" (UID: \"63a33f01-342a-4908-a744-795debfc5de4\") " pod="openshift-network-operator/iptables-alerter-cb4gp" Sep 16 00:34:11.984458 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984451 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-host\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:11.984966 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984481 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdrmx\" (UniqueName: \"kubernetes.io/projected/4f9387ae-f884-4b06-a3a1-0568ce7399ad-kube-api-access-sdrmx\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:11.984966 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984508 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dbzh\" (UniqueName: \"kubernetes.io/projected/ca719833-aa0c-4fe5-8aa4-019e479b8e10-kube-api-access-2dbzh\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:11.984966 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984532 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trnvd\" (UniqueName: \"kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd\") pod \"network-check-target-gw589\" (UID: \"f6c0e1cf-272e-412e-b846-d6ee5fc96e28\") " pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:11.984966 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984555 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca719833-aa0c-4fe5-8aa4-019e479b8e10-etc-selinux\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:11.984966 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984592 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/63a33f01-342a-4908-a744-795debfc5de4-iptables-alerter-script\") pod \"iptables-alerter-cb4gp\" (UID: \"63a33f01-342a-4908-a744-795debfc5de4\") " pod="openshift-network-operator/iptables-alerter-cb4gp" Sep 16 00:34:11.984966 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984616 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-modprobe-d\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:11.984966 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984637 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-run\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:11.984966 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984659 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-sys\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:11.984966 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984681 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f9387ae-f884-4b06-a3a1-0568ce7399ad-tmp\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:11.984966 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984702 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-var-lib-kubelet\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:11.984966 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984725 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca719833-aa0c-4fe5-8aa4-019e479b8e10-socket-dir\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:11.984966 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984760 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca719833-aa0c-4fe5-8aa4-019e479b8e10-registration-dir\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:11.984966 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984794 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1cd8f1ba-a180-412d-ac9b-8853de8af7ae-konnectivity-ca\") pod \"konnectivity-agent-jzl25\" (UID: \"1cd8f1ba-a180-412d-ac9b-8853de8af7ae\") " pod="kube-system/konnectivity-agent-jzl25" Sep 16 00:34:11.984966 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984817 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-sysctl-conf\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:11.984966 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984848 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgbjq\" (UniqueName: \"kubernetes.io/projected/63a33f01-342a-4908-a744-795debfc5de4-kube-api-access-kgbjq\") pod \"iptables-alerter-cb4gp\" (UID: \"63a33f01-342a-4908-a744-795debfc5de4\") " pod="openshift-network-operator/iptables-alerter-cb4gp" Sep 16 00:34:11.984966 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.984884 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-kubernetes\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:11.986436 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.986414 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 00:34:11.986540 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.986459 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 00:34:11.986540 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.986531 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 00:34:11.986653 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.986422 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-95m8b\"" Sep 16 00:34:11.986887 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.986719 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 00:34:11.986887 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.986793 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 00:34:11.986887 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.986854 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 00:34:11.987062 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.986908 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:11.988919 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.988902 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bglnh" Sep 16 00:34:11.993333 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.993316 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 00:34:11.993411 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.993332 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 00:34:11.993610 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.993593 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 00:34:11.993694 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.993682 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 00:34:11.993758 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.993741 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 00:34:11.994051 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.993913 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6wjhh\"" Sep 16 00:34:11.994051 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.993957 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kg2sp\"" Sep 16 00:34:11.994051 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:11.994025 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 00:34:12.005208 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.005191 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7np9l" Sep 16 00:34:12.012648 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.012631 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7np9l" Sep 16 00:34:12.072941 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.071398 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 00:34:12.072941 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.071660 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 00:34:12.085180 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085147 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-host-var-lib-cni-multus\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.085180 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085178 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-multus-conf-dir\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.085328 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085205 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-host-run-multus-certs\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.085328 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085236 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-log-socket\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.085328 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085308 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-run-ovn-kubernetes\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.085460 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085342 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-cni-netd\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.085460 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085366 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-env-overrides\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.085460 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085394 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63a33f01-342a-4908-a744-795debfc5de4-host-slash\") pod \"iptables-alerter-cb4gp\" (UID: \"63a33f01-342a-4908-a744-795debfc5de4\") " pod="openshift-network-operator/iptables-alerter-cb4gp" Sep 16 00:34:12.085460 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085417 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-systemd-units\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.085460 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085441 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8105e12c-26da-4f67-a85b-734fb527ad2b-cni-binary-copy\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.085460 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085456 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8105e12c-26da-4f67-a85b-734fb527ad2b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.085698 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085480 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-host\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.085698 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085515 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63a33f01-342a-4908-a744-795debfc5de4-host-slash\") pod \"iptables-alerter-cb4gp\" (UID: \"63a33f01-342a-4908-a744-795debfc5de4\") " pod="openshift-network-operator/iptables-alerter-cb4gp" Sep 16 00:34:12.085698 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085567 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdrmx\" (UniqueName: \"kubernetes.io/projected/4f9387ae-f884-4b06-a3a1-0568ce7399ad-kube-api-access-sdrmx\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.085698 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085602 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-host\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.085698 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085646 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe1a6aa-c150-47db-9b2f-e26f6415570f-tmp-dir\") pod \"node-resolver-cd4mj\" (UID: \"abe1a6aa-c150-47db-9b2f-e26f6415570f\") " pod="openshift-dns/node-resolver-cd4mj" Sep 16 00:34:12.085698 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085670 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be34ccf1-b7da-47ab-9134-2813a37e07a1-kubelet-config\") pod \"global-pull-secret-syncer-fpm24\" (UID: \"be34ccf1-b7da-47ab-9134-2813a37e07a1\") " pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:12.085882 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085768 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-kubelet\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.085882 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085792 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8105e12c-26da-4f67-a85b-734fb527ad2b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.085882 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085815 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dbzh\" (UniqueName: \"kubernetes.io/projected/ca719833-aa0c-4fe5-8aa4-019e479b8e10-kube-api-access-2dbzh\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:12.085882 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085834 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trnvd\" (UniqueName: \"kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd\") pod \"network-check-target-gw589\" (UID: \"f6c0e1cf-272e-412e-b846-d6ee5fc96e28\") " pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:12.085882 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085869 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca719833-aa0c-4fe5-8aa4-019e479b8e10-etc-selinux\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:12.086111 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085891 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-modprobe-d\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.086111 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085915 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs\") pod \"network-metrics-daemon-rxnhx\" (UID: \"b08b5ed1-8d6b-402b-88d2-5dbf09029651\") " pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:12.086111 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085937 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be34ccf1-b7da-47ab-9134-2813a37e07a1-dbus\") pod \"global-pull-secret-syncer-fpm24\" (UID: \"be34ccf1-b7da-47ab-9134-2813a37e07a1\") " pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:12.086111 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085957 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-run-openvswitch\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.086111 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085979 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1cd8f1ba-a180-412d-ac9b-8853de8af7ae-konnectivity-ca\") pod \"konnectivity-agent-jzl25\" (UID: \"1cd8f1ba-a180-412d-ac9b-8853de8af7ae\") " pod="kube-system/konnectivity-agent-jzl25" Sep 16 00:34:12.086111 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.085997 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d59da895-813d-4240-848d-9513b5886d1c-serviceca\") pod \"node-ca-q86vf\" (UID: \"d59da895-813d-4240-848d-9513b5886d1c\") " pod="openshift-image-registry/node-ca-q86vf" Sep 16 00:34:12.086111 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086020 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-ovn-node-metrics-cert\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.086111 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086052 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca719833-aa0c-4fe5-8aa4-019e479b8e10-etc-selinux\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:12.086111 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086061 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-modprobe-d\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.086111 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086073 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cctvm\" (UniqueName: \"kubernetes.io/projected/8105e12c-26da-4f67-a85b-734fb527ad2b-kube-api-access-cctvm\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.086111 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086097 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-host-var-lib-cni-bin\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.086550 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086119 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-run-ovn\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.086550 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086143 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-kubernetes\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.086550 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086164 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-systemd\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.086550 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086185 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5885041-68cc-4ec8-85e9-9555d7baa6c9-cni-binary-copy\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.086550 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086207 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-sysctl-d\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.086550 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086230 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca719833-aa0c-4fe5-8aa4-019e479b8e10-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:12.086550 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086261 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca719833-aa0c-4fe5-8aa4-019e479b8e10-device-dir\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:12.086550 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086286 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1cd8f1ba-a180-412d-ac9b-8853de8af7ae-agent-certs\") pod \"konnectivity-agent-jzl25\" (UID: \"1cd8f1ba-a180-412d-ac9b-8853de8af7ae\") " pod="kube-system/konnectivity-agent-jzl25" Sep 16 00:34:12.086550 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086308 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-multus-socket-dir-parent\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.086550 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086331 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-var-lib-openvswitch\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.086550 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086369 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8105e12c-26da-4f67-a85b-734fb527ad2b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.086550 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086395 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-sysconfig\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.086550 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086418 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-multus-cni-dir\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.086550 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086440 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-host-run-k8s-cni-cncf-io\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.086550 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086463 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-cni-bin\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.086550 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086486 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dj4d\" (UniqueName: \"kubernetes.io/projected/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-kube-api-access-6dj4d\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.086550 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086517 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-etc-kubernetes\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.087292 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086522 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-sysctl-d\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.087292 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086539 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8105e12c-26da-4f67-a85b-734fb527ad2b-cnibin\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.087292 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086563 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bn8d\" (UniqueName: \"kubernetes.io/projected/b08b5ed1-8d6b-402b-88d2-5dbf09029651-kube-api-access-6bn8d\") pod \"network-metrics-daemon-rxnhx\" (UID: \"b08b5ed1-8d6b-402b-88d2-5dbf09029651\") " pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:12.087292 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086587 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-host-var-lib-kubelet\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.087292 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086610 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-node-log\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.087292 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086617 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-kubernetes\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.087292 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086624 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1cd8f1ba-a180-412d-ac9b-8853de8af7ae-konnectivity-ca\") pod \"konnectivity-agent-jzl25\" (UID: \"1cd8f1ba-a180-412d-ac9b-8853de8af7ae\") " pod="kube-system/konnectivity-agent-jzl25" Sep 16 00:34:12.087292 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086653 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/63a33f01-342a-4908-a744-795debfc5de4-iptables-alerter-script\") pod \"iptables-alerter-cb4gp\" (UID: \"63a33f01-342a-4908-a744-795debfc5de4\") " pod="openshift-network-operator/iptables-alerter-cb4gp" Sep 16 00:34:12.087292 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086669 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-systemd\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.087292 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086676 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-run\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.087292 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086708 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-sys\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.087292 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086741 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca719833-aa0c-4fe5-8aa4-019e479b8e10-device-dir\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:12.087292 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086748 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f9387ae-f884-4b06-a3a1-0568ce7399ad-tmp\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.087292 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086773 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca719833-aa0c-4fe5-8aa4-019e479b8e10-socket-dir\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:12.087292 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086799 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-run\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.087292 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086798 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5885041-68cc-4ec8-85e9-9555d7baa6c9-multus-daemon-config\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.087292 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086838 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-var-lib-kubelet\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.088001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086858 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-sysconfig\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.088001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086864 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca719833-aa0c-4fe5-8aa4-019e479b8e10-registration-dir\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:12.088001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086915 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca719833-aa0c-4fe5-8aa4-019e479b8e10-registration-dir\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:12.088001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086918 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-var-lib-kubelet\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.088001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086957 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-sys\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.088001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.086985 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-system-cni-dir\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.088001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087009 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g8sl\" (UniqueName: \"kubernetes.io/projected/d5885041-68cc-4ec8-85e9-9555d7baa6c9-kube-api-access-5g8sl\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.088001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087032 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca719833-aa0c-4fe5-8aa4-019e479b8e10-socket-dir\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:12.088001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087060 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-etc-openvswitch\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.088001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087051 2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 00:34:12.088001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087137 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca719833-aa0c-4fe5-8aa4-019e479b8e10-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:12.088001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087166 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-ovnkube-config\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.088001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087189 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8105e12c-26da-4f67-a85b-734fb527ad2b-system-cni-dir\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.088001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087214 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-sysctl-conf\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.088001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087238 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/abe1a6aa-c150-47db-9b2f-e26f6415570f-hosts-file\") pod \"node-resolver-cd4mj\" (UID: \"abe1a6aa-c150-47db-9b2f-e26f6415570f\") " pod="openshift-dns/node-resolver-cd4mj" Sep 16 00:34:12.088001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087260 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret\") pod \"global-pull-secret-syncer-fpm24\" (UID: \"be34ccf1-b7da-47ab-9134-2813a37e07a1\") " pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:12.088001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087281 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-cnibin\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.088767 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087304 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.088767 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087332 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbjq\" (UniqueName: \"kubernetes.io/projected/63a33f01-342a-4908-a744-795debfc5de4-kube-api-access-kgbjq\") pod \"iptables-alerter-cb4gp\" (UID: \"63a33f01-342a-4908-a744-795debfc5de4\") " pod="openshift-network-operator/iptables-alerter-cb4gp" Sep 16 00:34:12.088767 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087356 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xjcf\" (UniqueName: \"kubernetes.io/projected/abe1a6aa-c150-47db-9b2f-e26f6415570f-kube-api-access-2xjcf\") pod \"node-resolver-cd4mj\" (UID: \"abe1a6aa-c150-47db-9b2f-e26f6415570f\") " pod="openshift-dns/node-resolver-cd4mj" Sep 16 00:34:12.088767 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087378 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smwsl\" (UniqueName: \"kubernetes.io/projected/d59da895-813d-4240-848d-9513b5886d1c-kube-api-access-smwsl\") pod \"node-ca-q86vf\" (UID: \"d59da895-813d-4240-848d-9513b5886d1c\") " pod="openshift-image-registry/node-ca-q86vf" Sep 16 00:34:12.088767 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087403 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-host-run-netns\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.088767 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087425 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-hostroot\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.088767 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087451 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-ovnkube-script-lib\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.088767 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087475 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca719833-aa0c-4fe5-8aa4-019e479b8e10-sys-fs\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:12.088767 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087500 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-tuned\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.088767 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087522 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d59da895-813d-4240-848d-9513b5886d1c-host\") pod \"node-ca-q86vf\" (UID: \"d59da895-813d-4240-848d-9513b5886d1c\") " pod="openshift-image-registry/node-ca-q86vf" Sep 16 00:34:12.088767 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087545 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/63a33f01-342a-4908-a744-795debfc5de4-iptables-alerter-script\") pod \"iptables-alerter-cb4gp\" (UID: \"63a33f01-342a-4908-a744-795debfc5de4\") " pod="openshift-network-operator/iptables-alerter-cb4gp" Sep 16 00:34:12.088767 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087546 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-slash\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.088767 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087582 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-run-netns\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.088767 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087602 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-run-systemd\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.088767 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087620 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-lib-modules\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.088767 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087674 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-sysctl-conf\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.088767 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087763 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f9387ae-f884-4b06-a3a1-0568ce7399ad-lib-modules\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.089631 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.087829 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca719833-aa0c-4fe5-8aa4-019e479b8e10-sys-fs\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:12.090736 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.090627 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f9387ae-f884-4b06-a3a1-0568ce7399ad-etc-tuned\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.090736 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.090690 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f9387ae-f884-4b06-a3a1-0568ce7399ad-tmp\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.090923 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.090905 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1cd8f1ba-a180-412d-ac9b-8853de8af7ae-agent-certs\") pod \"konnectivity-agent-jzl25\" (UID: \"1cd8f1ba-a180-412d-ac9b-8853de8af7ae\") " pod="kube-system/konnectivity-agent-jzl25" Sep 16 00:34:12.093621 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.093601 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdrmx\" (UniqueName: \"kubernetes.io/projected/4f9387ae-f884-4b06-a3a1-0568ce7399ad-kube-api-access-sdrmx\") pod \"tuned-26586\" (UID: \"4f9387ae-f884-4b06-a3a1-0568ce7399ad\") " pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.093880 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.093837 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dbzh\" (UniqueName: \"kubernetes.io/projected/ca719833-aa0c-4fe5-8aa4-019e479b8e10-kube-api-access-2dbzh\") pod \"aws-ebs-csi-driver-node-lm9zb\" (UID: \"ca719833-aa0c-4fe5-8aa4-019e479b8e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:12.095139 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:12.095119 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 00:34:12.095236 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:12.095147 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 00:34:12.095236 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:12.095159 2567 projected.go:196] Error preparing data for projected volume kube-api-access-trnvd for pod openshift-network-diagnostics/network-check-target-gw589: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 00:34:12.095236 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:12.095221 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd podName:f6c0e1cf-272e-412e-b846-d6ee5fc96e28 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:12.595202409 +0000 UTC m=+2.957425934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-trnvd" (UniqueName: "kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd") pod "network-check-target-gw589" (UID: "f6c0e1cf-272e-412e-b846-d6ee5fc96e28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 00:34:12.097863 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.097846 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgbjq\" (UniqueName: \"kubernetes.io/projected/63a33f01-342a-4908-a744-795debfc5de4-kube-api-access-kgbjq\") pod \"iptables-alerter-cb4gp\" (UID: \"63a33f01-342a-4908-a744-795debfc5de4\") " pod="openshift-network-operator/iptables-alerter-cb4gp" Sep 16 00:34:12.188895 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.188858 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-system-cni-dir\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.189088 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.188902 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g8sl\" (UniqueName: \"kubernetes.io/projected/d5885041-68cc-4ec8-85e9-9555d7baa6c9-kube-api-access-5g8sl\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.189088 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.188930 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-etc-openvswitch\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.189088 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.188980 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-etc-openvswitch\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.189088 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.188980 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-system-cni-dir\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.189088 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189081 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-ovnkube-config\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.189346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189103 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8105e12c-26da-4f67-a85b-734fb527ad2b-system-cni-dir\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.189346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189119 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/abe1a6aa-c150-47db-9b2f-e26f6415570f-hosts-file\") pod \"node-resolver-cd4mj\" (UID: \"abe1a6aa-c150-47db-9b2f-e26f6415570f\") " pod="openshift-dns/node-resolver-cd4mj" Sep 16 00:34:12.189346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189138 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret\") pod \"global-pull-secret-syncer-fpm24\" (UID: \"be34ccf1-b7da-47ab-9134-2813a37e07a1\") " pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:12.189346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189178 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8105e12c-26da-4f67-a85b-734fb527ad2b-system-cni-dir\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.189346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189180 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-cnibin\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.189346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189211 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.189346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189205 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/abe1a6aa-c150-47db-9b2f-e26f6415570f-hosts-file\") pod \"node-resolver-cd4mj\" (UID: \"abe1a6aa-c150-47db-9b2f-e26f6415570f\") " pod="openshift-dns/node-resolver-cd4mj" Sep 16 00:34:12.189346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189228 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-cnibin\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.189346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189237 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xjcf\" (UniqueName: \"kubernetes.io/projected/abe1a6aa-c150-47db-9b2f-e26f6415570f-kube-api-access-2xjcf\") pod \"node-resolver-cd4mj\" (UID: \"abe1a6aa-c150-47db-9b2f-e26f6415570f\") " pod="openshift-dns/node-resolver-cd4mj" Sep 16 00:34:12.189346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189262 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.189346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189262 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smwsl\" (UniqueName: \"kubernetes.io/projected/d59da895-813d-4240-848d-9513b5886d1c-kube-api-access-smwsl\") pod \"node-ca-q86vf\" (UID: \"d59da895-813d-4240-848d-9513b5886d1c\") " pod="openshift-image-registry/node-ca-q86vf" Sep 16 00:34:12.189346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189301 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-host-run-netns\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.189346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189325 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-hostroot\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.189346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189341 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-ovnkube-script-lib\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.189346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189346 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-host-run-netns\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.189346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189358 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d59da895-813d-4240-848d-9513b5886d1c-host\") pod \"node-ca-q86vf\" (UID: \"d59da895-813d-4240-848d-9513b5886d1c\") " pod="openshift-image-registry/node-ca-q86vf" Sep 16 00:34:12.190119 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189388 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-hostroot\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.190119 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189414 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-slash\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.190119 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189438 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-run-netns\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.190119 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189461 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-run-systemd\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.190119 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189465 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-slash\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.190119 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189486 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-host-var-lib-cni-multus\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.190119 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:12.189302 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 00:34:12.190119 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189519 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-multus-conf-dir\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.190119 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189526 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d59da895-813d-4240-848d-9513b5886d1c-host\") pod \"node-ca-q86vf\" (UID: \"d59da895-813d-4240-848d-9513b5886d1c\") " pod="openshift-image-registry/node-ca-q86vf" Sep 16 00:34:12.190119 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189533 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-run-netns\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.190119 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189531 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-run-systemd\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.190119 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189545 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-host-run-multus-certs\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.190119 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189564 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-host-var-lib-cni-multus\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.190119 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189577 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-host-run-multus-certs\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.190119 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189578 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-log-socket\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.190119 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189597 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-log-socket\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.190119 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:12.189602 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret podName:be34ccf1-b7da-47ab-9134-2813a37e07a1 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:12.689580048 +0000 UTC m=+3.051803575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret") pod "global-pull-secret-syncer-fpm24" (UID: "be34ccf1-b7da-47ab-9134-2813a37e07a1") : object "kube-system"/"original-pull-secret" not registered Sep 16 00:34:12.190119 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189610 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-multus-conf-dir\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.190952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189638 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-run-ovn-kubernetes\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.190952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189658 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-cni-netd\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.190952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189682 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-env-overrides\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.190952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189692 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-run-ovn-kubernetes\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.190952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189721 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-cni-netd\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.190952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189733 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-ovnkube-config\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.190952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189762 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-systemd-units\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.190952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189792 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8105e12c-26da-4f67-a85b-734fb527ad2b-cni-binary-copy\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.190952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189811 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-ovnkube-script-lib\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.190952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189825 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8105e12c-26da-4f67-a85b-734fb527ad2b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.190952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189854 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-systemd-units\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.190952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189860 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe1a6aa-c150-47db-9b2f-e26f6415570f-tmp-dir\") pod \"node-resolver-cd4mj\" (UID: \"abe1a6aa-c150-47db-9b2f-e26f6415570f\") " pod="openshift-dns/node-resolver-cd4mj" Sep 16 00:34:12.190952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189902 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be34ccf1-b7da-47ab-9134-2813a37e07a1-kubelet-config\") pod \"global-pull-secret-syncer-fpm24\" (UID: \"be34ccf1-b7da-47ab-9134-2813a37e07a1\") " pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:12.190952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189929 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-kubelet\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.190952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189955 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8105e12c-26da-4f67-a85b-734fb527ad2b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.190952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189998 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs\") pod \"network-metrics-daemon-rxnhx\" (UID: \"b08b5ed1-8d6b-402b-88d2-5dbf09029651\") " pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:12.190952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190023 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be34ccf1-b7da-47ab-9134-2813a37e07a1-dbus\") pod \"global-pull-secret-syncer-fpm24\" (UID: \"be34ccf1-b7da-47ab-9134-2813a37e07a1\") " pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:12.191633 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190031 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-kubelet\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.191633 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.189999 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8105e12c-26da-4f67-a85b-734fb527ad2b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.191633 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190068 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-run-openvswitch\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.191633 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190101 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d59da895-813d-4240-848d-9513b5886d1c-serviceca\") pod \"node-ca-q86vf\" (UID: \"d59da895-813d-4240-848d-9513b5886d1c\") " pod="openshift-image-registry/node-ca-q86vf" Sep 16 00:34:12.191633 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190107 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-run-openvswitch\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.191633 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190074 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-env-overrides\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.191633 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190126 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-ovn-node-metrics-cert\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.191633 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190163 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe1a6aa-c150-47db-9b2f-e26f6415570f-tmp-dir\") pod \"node-resolver-cd4mj\" (UID: \"abe1a6aa-c150-47db-9b2f-e26f6415570f\") " pod="openshift-dns/node-resolver-cd4mj" Sep 16 00:34:12.191633 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:12.190164 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 00:34:12.191633 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190198 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cctvm\" (UniqueName: \"kubernetes.io/projected/8105e12c-26da-4f67-a85b-734fb527ad2b-kube-api-access-cctvm\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.191633 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190226 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-host-var-lib-cni-bin\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.191633 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190234 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be34ccf1-b7da-47ab-9134-2813a37e07a1-dbus\") pod \"global-pull-secret-syncer-fpm24\" (UID: \"be34ccf1-b7da-47ab-9134-2813a37e07a1\") " pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:12.191633 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190260 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-host-var-lib-cni-bin\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.191633 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:12.190239 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs podName:b08b5ed1-8d6b-402b-88d2-5dbf09029651 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:12.690223525 +0000 UTC m=+3.052447039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs") pod "network-metrics-daemon-rxnhx" (UID: "b08b5ed1-8d6b-402b-88d2-5dbf09029651") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 00:34:12.191633 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190296 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-run-ovn\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.191633 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190316 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5885041-68cc-4ec8-85e9-9555d7baa6c9-cni-binary-copy\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.191633 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190345 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-multus-socket-dir-parent\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.192221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190371 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-var-lib-openvswitch\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.192221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190377 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8105e12c-26da-4f67-a85b-734fb527ad2b-cni-binary-copy\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.192221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190394 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8105e12c-26da-4f67-a85b-734fb527ad2b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.192221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190348 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-run-ovn\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.192221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190420 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be34ccf1-b7da-47ab-9134-2813a37e07a1-kubelet-config\") pod \"global-pull-secret-syncer-fpm24\" (UID: \"be34ccf1-b7da-47ab-9134-2813a37e07a1\") " pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:12.192221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190426 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-multus-cni-dir\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.192221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190460 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-multus-socket-dir-parent\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.192221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190469 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-host-run-k8s-cni-cncf-io\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.192221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190481 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-var-lib-openvswitch\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.192221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190496 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-cni-bin\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.192221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190520 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dj4d\" (UniqueName: \"kubernetes.io/projected/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-kube-api-access-6dj4d\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.192221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190539 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d59da895-813d-4240-848d-9513b5886d1c-serviceca\") pod \"node-ca-q86vf\" (UID: \"d59da895-813d-4240-848d-9513b5886d1c\") " pod="openshift-image-registry/node-ca-q86vf" Sep 16 00:34:12.192221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190551 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-host-cni-bin\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.192221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190471 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-multus-cni-dir\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.192221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190544 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-etc-kubernetes\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.192221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190573 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-etc-kubernetes\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.192221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190519 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-host-run-k8s-cni-cncf-io\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.192897 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190594 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8105e12c-26da-4f67-a85b-734fb527ad2b-cnibin\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.192897 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190622 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bn8d\" (UniqueName: \"kubernetes.io/projected/b08b5ed1-8d6b-402b-88d2-5dbf09029651-kube-api-access-6bn8d\") pod \"network-metrics-daemon-rxnhx\" (UID: \"b08b5ed1-8d6b-402b-88d2-5dbf09029651\") " pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:12.192897 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190626 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8105e12c-26da-4f67-a85b-734fb527ad2b-cnibin\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.192897 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190646 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-host-var-lib-kubelet\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.192897 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190671 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-node-log\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.192897 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190702 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5885041-68cc-4ec8-85e9-9555d7baa6c9-multus-daemon-config\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.192897 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190732 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5885041-68cc-4ec8-85e9-9555d7baa6c9-host-var-lib-kubelet\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.192897 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190786 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-node-log\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.192897 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.190909 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5885041-68cc-4ec8-85e9-9555d7baa6c9-cni-binary-copy\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.192897 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.191092 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8105e12c-26da-4f67-a85b-734fb527ad2b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.192897 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.191140 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8105e12c-26da-4f67-a85b-734fb527ad2b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.192897 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.191309 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5885041-68cc-4ec8-85e9-9555d7baa6c9-multus-daemon-config\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.192897 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.192837 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-ovn-node-metrics-cert\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.199059 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.199015 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cctvm\" (UniqueName: \"kubernetes.io/projected/8105e12c-26da-4f67-a85b-734fb527ad2b-kube-api-access-cctvm\") pod \"multus-additional-cni-plugins-9spsl\" (UID: \"8105e12c-26da-4f67-a85b-734fb527ad2b\") " pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.199771 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.199728 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dj4d\" (UniqueName: \"kubernetes.io/projected/3eefce2f-3b96-4b3b-ba40-4d9891bb457e-kube-api-access-6dj4d\") pod \"ovnkube-node-75ccc\" (UID: \"3eefce2f-3b96-4b3b-ba40-4d9891bb457e\") " pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.201342 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.200328 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smwsl\" (UniqueName: \"kubernetes.io/projected/d59da895-813d-4240-848d-9513b5886d1c-kube-api-access-smwsl\") pod \"node-ca-q86vf\" (UID: \"d59da895-813d-4240-848d-9513b5886d1c\") " pod="openshift-image-registry/node-ca-q86vf" Sep 16 00:34:12.203501 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.203468 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g8sl\" (UniqueName: \"kubernetes.io/projected/d5885041-68cc-4ec8-85e9-9555d7baa6c9-kube-api-access-5g8sl\") pod \"multus-bglnh\" (UID: \"d5885041-68cc-4ec8-85e9-9555d7baa6c9\") " pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.203501 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.203495 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xjcf\" (UniqueName: \"kubernetes.io/projected/abe1a6aa-c150-47db-9b2f-e26f6415570f-kube-api-access-2xjcf\") pod \"node-resolver-cd4mj\" (UID: \"abe1a6aa-c150-47db-9b2f-e26f6415570f\") " pod="openshift-dns/node-resolver-cd4mj" Sep 16 00:34:12.203696 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.203676 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bn8d\" (UniqueName: \"kubernetes.io/projected/b08b5ed1-8d6b-402b-88d2-5dbf09029651-kube-api-access-6bn8d\") pod \"network-metrics-daemon-rxnhx\" (UID: \"b08b5ed1-8d6b-402b-88d2-5dbf09029651\") " pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:12.273908 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.273882 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" Sep 16 00:34:12.283744 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.283715 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jzl25" Sep 16 00:34:12.296830 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.296811 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cb4gp" Sep 16 00:34:12.298676 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.298652 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-26586" Sep 16 00:34:12.304693 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.304678 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cd4mj" Sep 16 00:34:12.311746 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.311729 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q86vf" Sep 16 00:34:12.317922 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.317905 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:12.323143 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.323127 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9spsl" Sep 16 00:34:12.331367 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.331352 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bglnh" Sep 16 00:34:12.538066 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.537974 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 00:34:12.695161 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.695123 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trnvd\" (UniqueName: \"kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd\") pod \"network-check-target-gw589\" (UID: \"f6c0e1cf-272e-412e-b846-d6ee5fc96e28\") " pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:12.695161 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.695156 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs\") pod \"network-metrics-daemon-rxnhx\" (UID: \"b08b5ed1-8d6b-402b-88d2-5dbf09029651\") " pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:12.695395 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:12.695268 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 00:34:12.695395 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:12.695295 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret\") pod \"global-pull-secret-syncer-fpm24\" (UID: \"be34ccf1-b7da-47ab-9134-2813a37e07a1\") " pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:12.695395 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:12.695303 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 00:34:12.695395 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:12.695316 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs podName:b08b5ed1-8d6b-402b-88d2-5dbf09029651 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:13.695303267 +0000 UTC m=+4.057526777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs") pod "network-metrics-daemon-rxnhx" (UID: "b08b5ed1-8d6b-402b-88d2-5dbf09029651") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 00:34:12.695395 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:12.695331 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 00:34:12.695395 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:12.695344 2567 projected.go:196] Error preparing data for projected volume kube-api-access-trnvd for pod openshift-network-diagnostics/network-check-target-gw589: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 00:34:12.695395 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:12.695357 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 00:34:12.695395 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:12.695398 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret podName:be34ccf1-b7da-47ab-9134-2813a37e07a1 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:13.695382776 +0000 UTC m=+4.057606285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret") pod "global-pull-secret-syncer-fpm24" (UID: "be34ccf1-b7da-47ab-9134-2813a37e07a1") : object "kube-system"/"original-pull-secret" not registered Sep 16 00:34:12.695789 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:12.695415 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd podName:f6c0e1cf-272e-412e-b846-d6ee5fc96e28 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:13.695405816 +0000 UTC m=+4.057629328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-trnvd" (UniqueName: "kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd") pod "network-check-target-gw589" (UID: "f6c0e1cf-272e-412e-b846-d6ee5fc96e28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 00:34:12.985451 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:12.985415 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eefce2f_3b96_4b3b_ba40_4d9891bb457e.slice/crio-d1617973b66f26f4948eddd36623bd126b6bdb2ee453e71407eec8e354f843be WatchSource:0}: Error finding container d1617973b66f26f4948eddd36623bd126b6bdb2ee453e71407eec8e354f843be: Status 404 returned error can't find the container with id d1617973b66f26f4948eddd36623bd126b6bdb2ee453e71407eec8e354f843be Sep 16 00:34:12.986469 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:12.986445 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8105e12c_26da_4f67_a85b_734fb527ad2b.slice/crio-7c276d92bd90a8dded74124a0453689bff735048d87987edf59c272174539699 WatchSource:0}: Error finding container 7c276d92bd90a8dded74124a0453689bff735048d87987edf59c272174539699: Status 404 returned error can't find the container with id 7c276d92bd90a8dded74124a0453689bff735048d87987edf59c272174539699 Sep 16 00:34:12.987713 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:12.987690 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59da895_813d_4240_848d_9513b5886d1c.slice/crio-75d9479aa69d7ab3a01233e2cd90aa51f4b106fc3e7eebd6d2c7093212aacee0 WatchSource:0}: Error finding container 75d9479aa69d7ab3a01233e2cd90aa51f4b106fc3e7eebd6d2c7093212aacee0: Status 404 returned error can't find the container with id 75d9479aa69d7ab3a01233e2cd90aa51f4b106fc3e7eebd6d2c7093212aacee0 Sep 16 00:34:12.988764 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:12.988732 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe1a6aa_c150_47db_9b2f_e26f6415570f.slice/crio-6b9f88b87a74d5b875e9e6c8fd3ff990bd80e7031df0d5e7206cb35674713d8e WatchSource:0}: Error finding container 6b9f88b87a74d5b875e9e6c8fd3ff990bd80e7031df0d5e7206cb35674713d8e: Status 404 returned error can't find the container with id 6b9f88b87a74d5b875e9e6c8fd3ff990bd80e7031df0d5e7206cb35674713d8e Sep 16 00:34:12.991998 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:12.991974 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5885041_68cc_4ec8_85e9_9555d7baa6c9.slice/crio-e234f85167c92dd909bb919b19425ffe97e793a6682a1598be3d8e64e1212d71 WatchSource:0}: Error finding container e234f85167c92dd909bb919b19425ffe97e793a6682a1598be3d8e64e1212d71: Status 404 returned error can't find the container with id e234f85167c92dd909bb919b19425ffe97e793a6682a1598be3d8e64e1212d71 Sep 16 00:34:12.993026 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:12.993002 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a33f01_342a_4908_a744_795debfc5de4.slice/crio-9f0f480fad92d5f84aa0d6c79b490c5588ef8e86609eafac7cfb454f0f4a0b2f WatchSource:0}: Error finding container 9f0f480fad92d5f84aa0d6c79b490c5588ef8e86609eafac7cfb454f0f4a0b2f: Status 404 returned error can't find the container with id 9f0f480fad92d5f84aa0d6c79b490c5588ef8e86609eafac7cfb454f0f4a0b2f Sep 16 00:34:12.994023 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:12.993985 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd8f1ba_a180_412d_ac9b_8853de8af7ae.slice/crio-46fdf20d90bdc82446ed3f6dce1dadc73fb1586810df9d38e4d0a154127d9aae WatchSource:0}: Error finding container 46fdf20d90bdc82446ed3f6dce1dadc73fb1586810df9d38e4d0a154127d9aae: Status 404 returned error can't find the container with id 46fdf20d90bdc82446ed3f6dce1dadc73fb1586810df9d38e4d0a154127d9aae Sep 16 00:34:12.998064 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:12.997447 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca719833_aa0c_4fe5_8aa4_019e479b8e10.slice/crio-a7ab2f3bd4380009ac330d98c8c83cbe1fc93bc37443820f24ae8fba3d6443d5 WatchSource:0}: Error finding container a7ab2f3bd4380009ac330d98c8c83cbe1fc93bc37443820f24ae8fba3d6443d5: Status 404 returned error can't find the container with id a7ab2f3bd4380009ac330d98c8c83cbe1fc93bc37443820f24ae8fba3d6443d5 Sep 16 00:34:13.016424 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:13.016379 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 00:29:12 +0000 UTC" deadline="2028-05-17 05:57:33.016486488 +0000 UTC" Sep 16 00:34:13.016424 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:13.016419 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14621h23m20.000069675s" Sep 16 00:34:13.113183 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:13.113159 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:13.113342 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:13.113317 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:13.127937 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:13.127915 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-26586" event={"ID":"4f9387ae-f884-4b06-a3a1-0568ce7399ad","Type":"ContainerStarted","Data":"32c4ec584e5cac6e2fa1b6131c6fab9f28416769d242036996b5b0daa8856e80"} Sep 16 00:34:13.129370 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:13.129347 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" event={"ID":"ca719833-aa0c-4fe5-8aa4-019e479b8e10","Type":"ContainerStarted","Data":"a7ab2f3bd4380009ac330d98c8c83cbe1fc93bc37443820f24ae8fba3d6443d5"} Sep 16 00:34:13.131136 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:13.131104 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cb4gp" event={"ID":"63a33f01-342a-4908-a744-795debfc5de4","Type":"ContainerStarted","Data":"9f0f480fad92d5f84aa0d6c79b490c5588ef8e86609eafac7cfb454f0f4a0b2f"} Sep 16 00:34:13.133104 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:13.133081 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jzl25" event={"ID":"1cd8f1ba-a180-412d-ac9b-8853de8af7ae","Type":"ContainerStarted","Data":"46fdf20d90bdc82446ed3f6dce1dadc73fb1586810df9d38e4d0a154127d9aae"} Sep 16 00:34:13.134432 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:13.134413 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bglnh" event={"ID":"d5885041-68cc-4ec8-85e9-9555d7baa6c9","Type":"ContainerStarted","Data":"e234f85167c92dd909bb919b19425ffe97e793a6682a1598be3d8e64e1212d71"} Sep 16 00:34:13.135386 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:13.135368 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cd4mj" event={"ID":"abe1a6aa-c150-47db-9b2f-e26f6415570f","Type":"ContainerStarted","Data":"6b9f88b87a74d5b875e9e6c8fd3ff990bd80e7031df0d5e7206cb35674713d8e"} Sep 16 00:34:13.137538 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:13.137518 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q86vf" event={"ID":"d59da895-813d-4240-848d-9513b5886d1c","Type":"ContainerStarted","Data":"75d9479aa69d7ab3a01233e2cd90aa51f4b106fc3e7eebd6d2c7093212aacee0"} Sep 16 00:34:13.142328 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:13.142308 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-194.ec2.internal" event={"ID":"67a2beadf904429fdc039f04a692727f","Type":"ContainerStarted","Data":"35e7a67ba90ace29e02431b1883cf43af773145ddea743b3cd589d60c8c12af2"} Sep 16 00:34:13.143331 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:13.143309 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9spsl" event={"ID":"8105e12c-26da-4f67-a85b-734fb527ad2b","Type":"ContainerStarted","Data":"7c276d92bd90a8dded74124a0453689bff735048d87987edf59c272174539699"} Sep 16 00:34:13.144884 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:13.144861 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" event={"ID":"3eefce2f-3b96-4b3b-ba40-4d9891bb457e","Type":"ContainerStarted","Data":"d1617973b66f26f4948eddd36623bd126b6bdb2ee453e71407eec8e354f843be"} Sep 16 00:34:13.155002 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:13.154963 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-194.ec2.internal" podStartSLOduration=2.154950971 podStartE2EDuration="2.154950971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:13.154565387 +0000 UTC m=+3.516788919" watchObservedRunningTime="2026-09-16 00:34:13.154950971 +0000 UTC m=+3.517174503" Sep 16 00:34:13.702890 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:13.702550 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret\") pod \"global-pull-secret-syncer-fpm24\" (UID: \"be34ccf1-b7da-47ab-9134-2813a37e07a1\") " pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:13.702890 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:13.702827 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 00:34:13.702890 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:13.702857 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trnvd\" (UniqueName: \"kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd\") pod \"network-check-target-gw589\" (UID: \"f6c0e1cf-272e-412e-b846-d6ee5fc96e28\") " pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:13.702890 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:13.702890 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs\") pod \"network-metrics-daemon-rxnhx\" (UID: \"b08b5ed1-8d6b-402b-88d2-5dbf09029651\") " pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:13.703239 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:13.702959 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret podName:be34ccf1-b7da-47ab-9134-2813a37e07a1 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:15.702940351 +0000 UTC m=+6.065163868 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret") pod "global-pull-secret-syncer-fpm24" (UID: "be34ccf1-b7da-47ab-9134-2813a37e07a1") : object "kube-system"/"original-pull-secret" not registered Sep 16 00:34:13.703239 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:13.703076 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 00:34:13.703239 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:13.703093 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 00:34:13.703239 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:13.703104 2567 projected.go:196] Error preparing data for projected volume kube-api-access-trnvd for pod openshift-network-diagnostics/network-check-target-gw589: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 00:34:13.703239 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:13.703149 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd podName:f6c0e1cf-272e-412e-b846-d6ee5fc96e28 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:15.703137545 +0000 UTC m=+6.065361069 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-trnvd" (UniqueName: "kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd") pod "network-check-target-gw589" (UID: "f6c0e1cf-272e-412e-b846-d6ee5fc96e28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 00:34:13.703239 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:13.703213 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 00:34:13.703539 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:13.703246 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs podName:b08b5ed1-8d6b-402b-88d2-5dbf09029651 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:15.703234002 +0000 UTC m=+6.065457515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs") pod "network-metrics-daemon-rxnhx" (UID: "b08b5ed1-8d6b-402b-88d2-5dbf09029651") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 00:34:14.017098 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:14.016971 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 00:29:12 +0000 UTC" deadline="2028-03-29 21:41:11.713972762 +0000 UTC" Sep 16 00:34:14.017098 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:14.017051 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13461h6m57.696943494s" Sep 16 00:34:14.115567 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:14.115054 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:14.115567 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:14.115239 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:14.115567 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:14.115377 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:14.115567 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:14.115482 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:14.153398 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:14.153364 2567 generic.go:356] "Generic (PLEG): container finished" podID="fa7d14c1ae359f30b74b1f397edcabc9" containerID="d03b0774f16bba470c5454ab0775b3025e3e4c042e93b1f3e77178f6b8bcee60" exitCode=0 Sep 16 00:34:14.153936 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:14.153913 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal" event={"ID":"fa7d14c1ae359f30b74b1f397edcabc9","Type":"ContainerDied","Data":"d03b0774f16bba470c5454ab0775b3025e3e4c042e93b1f3e77178f6b8bcee60"} Sep 16 00:34:15.112529 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:15.112493 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:15.112978 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:15.112639 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:15.181972 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:15.181935 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal" event={"ID":"fa7d14c1ae359f30b74b1f397edcabc9","Type":"ContainerStarted","Data":"39892736e9d662624bd5e900df8af2cc2bbb780320acad2b7367c38abc6926eb"} Sep 16 00:34:15.722806 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:15.721878 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs\") pod \"network-metrics-daemon-rxnhx\" (UID: \"b08b5ed1-8d6b-402b-88d2-5dbf09029651\") " pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:15.722806 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:15.721951 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret\") pod \"global-pull-secret-syncer-fpm24\" (UID: \"be34ccf1-b7da-47ab-9134-2813a37e07a1\") " pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:15.722806 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:15.721997 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trnvd\" (UniqueName: \"kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd\") pod \"network-check-target-gw589\" (UID: \"f6c0e1cf-272e-412e-b846-d6ee5fc96e28\") " pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:15.722806 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:15.722153 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 00:34:15.722806 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:15.722173 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 00:34:15.722806 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:15.722186 2567 projected.go:196] Error preparing data for projected volume kube-api-access-trnvd for pod openshift-network-diagnostics/network-check-target-gw589: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 00:34:15.722806 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:15.722241 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd podName:f6c0e1cf-272e-412e-b846-d6ee5fc96e28 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:19.722223541 +0000 UTC m=+10.084447057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-trnvd" (UniqueName: "kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd") pod "network-check-target-gw589" (UID: "f6c0e1cf-272e-412e-b846-d6ee5fc96e28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 00:34:15.722806 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:15.722635 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 00:34:15.722806 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:15.722685 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs podName:b08b5ed1-8d6b-402b-88d2-5dbf09029651 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:19.722670473 +0000 UTC m=+10.084893989 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs") pod "network-metrics-daemon-rxnhx" (UID: "b08b5ed1-8d6b-402b-88d2-5dbf09029651") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 00:34:15.722806 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:15.722745 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 00:34:15.722806 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:15.722775 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret podName:be34ccf1-b7da-47ab-9134-2813a37e07a1 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:19.722764997 +0000 UTC m=+10.084988513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret") pod "global-pull-secret-syncer-fpm24" (UID: "be34ccf1-b7da-47ab-9134-2813a37e07a1") : object "kube-system"/"original-pull-secret" not registered Sep 16 00:34:16.114234 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:16.114134 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:16.114669 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:16.114268 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:16.114669 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:16.114147 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:16.114830 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:16.114674 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:17.113700 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:17.113172 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:17.113700 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:17.113333 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:18.112504 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:18.112466 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:18.112949 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:18.112595 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:18.112949 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:18.112651 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:18.112949 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:18.112780 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:19.112490 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:19.112459 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:19.112657 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:19.112603 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:19.753820 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:19.753424 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret\") pod \"global-pull-secret-syncer-fpm24\" (UID: \"be34ccf1-b7da-47ab-9134-2813a37e07a1\") " pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:19.753820 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:19.753491 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trnvd\" (UniqueName: \"kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd\") pod \"network-check-target-gw589\" (UID: \"f6c0e1cf-272e-412e-b846-d6ee5fc96e28\") " pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:19.753820 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:19.753519 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs\") pod \"network-metrics-daemon-rxnhx\" (UID: \"b08b5ed1-8d6b-402b-88d2-5dbf09029651\") " pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:19.753820 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:19.753600 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 00:34:19.753820 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:19.753619 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 00:34:19.753820 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:19.753672 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret podName:be34ccf1-b7da-47ab-9134-2813a37e07a1 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:27.75365043 +0000 UTC m=+18.115873955 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret") pod "global-pull-secret-syncer-fpm24" (UID: "be34ccf1-b7da-47ab-9134-2813a37e07a1") : object "kube-system"/"original-pull-secret" not registered Sep 16 00:34:19.753820 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:19.753695 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs podName:b08b5ed1-8d6b-402b-88d2-5dbf09029651 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:27.75368354 +0000 UTC m=+18.115907052 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs") pod "network-metrics-daemon-rxnhx" (UID: "b08b5ed1-8d6b-402b-88d2-5dbf09029651") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 00:34:19.753820 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:19.753723 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 00:34:19.753820 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:19.753740 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 00:34:19.753820 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:19.753751 2567 projected.go:196] Error preparing data for projected volume kube-api-access-trnvd for pod openshift-network-diagnostics/network-check-target-gw589: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 00:34:19.753820 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:19.753789 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd podName:f6c0e1cf-272e-412e-b846-d6ee5fc96e28 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:27.753773158 +0000 UTC m=+18.115996674 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-trnvd" (UniqueName: "kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd") pod "network-check-target-gw589" (UID: "f6c0e1cf-272e-412e-b846-d6ee5fc96e28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 00:34:20.113566 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:20.113487 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:20.114005 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:20.113584 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:20.114005 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:20.113961 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:20.114147 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:20.114063 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:21.113302 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:21.113266 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:21.113485 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:21.113414 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:22.113240 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:22.113201 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:22.113662 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:22.113326 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:22.113831 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:22.113813 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:22.113915 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:22.113889 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:23.112800 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:23.112772 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:23.112980 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:23.112909 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:24.112800 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:24.112766 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:24.113184 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:24.112898 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:24.113184 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:24.112955 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:24.113184 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:24.113079 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:25.112626 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:25.112590 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:25.112795 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:25.112743 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:26.113332 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:26.113289 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:26.113768 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:26.113293 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:26.113768 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:26.113404 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:26.113768 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:26.113487 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:27.112606 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:27.112588 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:27.112705 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:27.112687 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:27.808325 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:27.808232 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trnvd\" (UniqueName: \"kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd\") pod \"network-check-target-gw589\" (UID: \"f6c0e1cf-272e-412e-b846-d6ee5fc96e28\") " pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:27.808325 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:27.808286 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs\") pod \"network-metrics-daemon-rxnhx\" (UID: \"b08b5ed1-8d6b-402b-88d2-5dbf09029651\") " pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:27.808773 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:27.808335 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret\") pod \"global-pull-secret-syncer-fpm24\" (UID: \"be34ccf1-b7da-47ab-9134-2813a37e07a1\") " pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:27.808773 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:27.808449 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 00:34:27.808773 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:27.808513 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret podName:be34ccf1-b7da-47ab-9134-2813a37e07a1 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:43.808494775 +0000 UTC m=+34.170718343 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret") pod "global-pull-secret-syncer-fpm24" (UID: "be34ccf1-b7da-47ab-9134-2813a37e07a1") : object "kube-system"/"original-pull-secret" not registered Sep 16 00:34:27.808941 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:27.808922 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 00:34:27.808992 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:27.808947 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 00:34:27.808992 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:27.808960 2567 projected.go:196] Error preparing data for projected volume kube-api-access-trnvd for pod openshift-network-diagnostics/network-check-target-gw589: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 00:34:27.809133 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:27.809006 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd podName:f6c0e1cf-272e-412e-b846-d6ee5fc96e28 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:43.808992238 +0000 UTC m=+34.171215750 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-trnvd" (UniqueName: "kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd") pod "network-check-target-gw589" (UID: "f6c0e1cf-272e-412e-b846-d6ee5fc96e28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 00:34:27.809133 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:27.809093 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 00:34:27.809133 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:27.809128 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs podName:b08b5ed1-8d6b-402b-88d2-5dbf09029651 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:43.809115488 +0000 UTC m=+34.171339001 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs") pod "network-metrics-daemon-rxnhx" (UID: "b08b5ed1-8d6b-402b-88d2-5dbf09029651") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 00:34:28.112766 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:28.112676 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:28.112910 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:28.112805 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:28.112910 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:28.112858 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:28.113010 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:28.112944 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:29.113006 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:29.112973 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:29.113425 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:29.113117 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:30.113851 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:30.113815 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:30.114294 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:30.113897 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:30.114294 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:30.113920 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:30.114294 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:30.114018 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:31.113231 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:31.113208 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:31.113327 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:31.113310 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:32.113589 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.113093 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:32.114610 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.113093 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:32.114610 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:32.113640 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:32.114610 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:32.113699 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:32.216972 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.216933 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-26586" event={"ID":"4f9387ae-f884-4b06-a3a1-0568ce7399ad","Type":"ContainerStarted","Data":"cc8ed320e9105cf65aefb5b10bd4f2b7c26768a94c7ddcf5914283c1b80d799f"} Sep 16 00:34:32.218444 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.218419 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" event={"ID":"ca719833-aa0c-4fe5-8aa4-019e479b8e10","Type":"ContainerStarted","Data":"a1d5130d6c602916e775dc608b64b30e4208a91f236013f742ddc79248035378"} Sep 16 00:34:32.219756 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.219733 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jzl25" event={"ID":"1cd8f1ba-a180-412d-ac9b-8853de8af7ae","Type":"ContainerStarted","Data":"a508abe2a919352edf1c67c4ff2e3466b93e5dafbcdc8a95d98c9dbec44df9fb"} Sep 16 00:34:32.221211 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.221191 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bglnh" event={"ID":"d5885041-68cc-4ec8-85e9-9555d7baa6c9","Type":"ContainerStarted","Data":"0a495923d5662168f70c81ef02e512cecc8267d2f9ed3a6514e062fc8aa64a80"} Sep 16 00:34:32.222582 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.222562 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cd4mj" event={"ID":"abe1a6aa-c150-47db-9b2f-e26f6415570f","Type":"ContainerStarted","Data":"415ec29a055d5d4882c54eced1f546e8cf59d5a03e5a0c7dd5d831f077f39d69"} Sep 16 00:34:32.223969 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.223942 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q86vf" event={"ID":"d59da895-813d-4240-848d-9513b5886d1c","Type":"ContainerStarted","Data":"6437f6447e4986703f5ca7e4832b70fa5fd59dac86c5d54685b396d882dabe5a"} Sep 16 00:34:32.225381 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.225361 2567 generic.go:356] "Generic (PLEG): container finished" podID="8105e12c-26da-4f67-a85b-734fb527ad2b" containerID="74451c2c68a6d488ca2fcff692c6691c5273381f6308b2395df2183af11accd7" exitCode=0 Sep 16 00:34:32.225465 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.225426 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9spsl" event={"ID":"8105e12c-26da-4f67-a85b-734fb527ad2b","Type":"ContainerDied","Data":"74451c2c68a6d488ca2fcff692c6691c5273381f6308b2395df2183af11accd7"} Sep 16 00:34:32.228408 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.228387 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" event={"ID":"3eefce2f-3b96-4b3b-ba40-4d9891bb457e","Type":"ContainerStarted","Data":"898ed974a2ac5187222fcd68f3c333238f03bc5325212d4cf8ba905ebde82406"} Sep 16 00:34:32.228488 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.228448 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" event={"ID":"3eefce2f-3b96-4b3b-ba40-4d9891bb457e","Type":"ContainerStarted","Data":"d25fbcfbaf738bd55ce469351974c8f539e9293181d0a6ee45130ab30ff8bcdf"} Sep 16 00:34:32.228488 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.228484 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" event={"ID":"3eefce2f-3b96-4b3b-ba40-4d9891bb457e","Type":"ContainerStarted","Data":"5f7e8beded8d49979090d8560f233cb6b025c04a625a30987e90763762b85135"} Sep 16 00:34:32.228564 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.228512 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" event={"ID":"3eefce2f-3b96-4b3b-ba40-4d9891bb457e","Type":"ContainerStarted","Data":"a883399f6fd2c6c17a1dbad559670c3f3cec88c2c264f991d08a80332f8dc692"} Sep 16 00:34:32.228564 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.228538 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" event={"ID":"3eefce2f-3b96-4b3b-ba40-4d9891bb457e","Type":"ContainerStarted","Data":"af608b12b0c10395d52b51bdac61a8932bb38e52974f770dd55eccc997cd842f"} Sep 16 00:34:32.228646 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.228572 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" event={"ID":"3eefce2f-3b96-4b3b-ba40-4d9891bb457e","Type":"ContainerStarted","Data":"b6623f9aad98e61e491c4c25b3bf4b7124b4c500337ff51d6da516cf5b86aadb"} Sep 16 00:34:32.233945 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.233910 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-26586" podStartSLOduration=4.092776335 podStartE2EDuration="22.233898271s" totalImagesPullingTime="18.141121936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:32.233616672 +0000 UTC m=+22.595840223" watchObservedRunningTime="2026-09-16 00:34:32.233898271 +0000 UTC m=+22.596121800" Sep 16 00:34:32.234088 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.234069 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-194.ec2.internal" podStartSLOduration=21.23406436 podStartE2EDuration="21.23406436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:15.198205318 +0000 UTC m=+5.560428855" watchObservedRunningTime="2026-09-16 00:34:32.23406436 +0000 UTC m=+22.596287889" Sep 16 00:34:32.265992 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.265950 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jzl25" podStartSLOduration=4.169149409 podStartE2EDuration="22.265938538s" totalImagesPullingTime="18.096789129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:32.251592987 +0000 UTC m=+22.613816518" watchObservedRunningTime="2026-09-16 00:34:32.265938538 +0000 UTC m=+22.628162069" Sep 16 00:34:32.266132 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.266114 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q86vf" podStartSLOduration=4.116842805 podStartE2EDuration="22.266108781s" totalImagesPullingTime="18.149265976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:32.265769763 +0000 UTC m=+22.627993295" watchObservedRunningTime="2026-09-16 00:34:32.266108781 +0000 UTC m=+22.628332313" Sep 16 00:34:32.310520 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.310478 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bglnh" podStartSLOduration=3.9674973749999998 podStartE2EDuration="22.310468409s" totalImagesPullingTime="18.342971034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:32.310224394 +0000 UTC m=+22.672447927" watchObservedRunningTime="2026-09-16 00:34:32.310468409 +0000 UTC m=+22.672691940" Sep 16 00:34:32.325333 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.325293 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cd4mj" podStartSLOduration=4.176584986 podStartE2EDuration="22.325282834s" totalImagesPullingTime="18.148697848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:32.324842536 +0000 UTC m=+22.687066069" watchObservedRunningTime="2026-09-16 00:34:32.325282834 +0000 UTC m=+22.687506366" Sep 16 00:34:32.350344 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.350325 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:32.429978 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.429955 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:32.484423 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.484398 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:32.530985 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.530949 2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 00:34:32.881331 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:32.881262 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:33.044177 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:33.044077 2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T00:34:32.530968914Z","UUID":"c306d216-4380-4fcd-aba2-2299ec9768a4","Handler":null,"Name":"","Endpoint":""} Sep 16 00:34:33.045770 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:33.045749 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 00:34:33.045897 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:33.045783 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 00:34:33.113251 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:33.113223 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:33.113400 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:33.113331 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:33.235348 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:33.235269 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" event={"ID":"ca719833-aa0c-4fe5-8aa4-019e479b8e10","Type":"ContainerStarted","Data":"a164990aacb1c345615ed358584ebe4099e66bcdc73a6a088ccbcacb5ad76f34"} Sep 16 00:34:33.237270 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:33.237222 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cb4gp" event={"ID":"63a33f01-342a-4908-a744-795debfc5de4","Type":"ContainerStarted","Data":"c0f68b966d596f12906b493678fe61b73569e8f5ecfd73fd7c92137fa2e12e67"} Sep 16 00:34:33.253599 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:33.253551 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cb4gp" podStartSLOduration=5.112155024 podStartE2EDuration="23.253536714s" totalImagesPullingTime="18.14138169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:33.253259817 +0000 UTC m=+23.615483369" watchObservedRunningTime="2026-09-16 00:34:33.253536714 +0000 UTC m=+23.615760246" Sep 16 00:34:33.367964 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:33.367927 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:33.497321 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:33.497248 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:33.597226 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:33.597196 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:34.113013 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:34.112937 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:34.113194 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:34.113056 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:34.113194 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:34.113091 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:34.113194 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:34.113146 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:34.163753 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:34.163712 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jzl25" Sep 16 00:34:34.164374 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:34.164348 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jzl25" Sep 16 00:34:34.242449 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:34.242417 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" event={"ID":"ca719833-aa0c-4fe5-8aa4-019e479b8e10","Type":"ContainerStarted","Data":"6dd0adf8d0e3b3829b8f70e68b3a21c2c996e9151befc8300534a8efb10f9175"} Sep 16 00:34:34.246562 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:34.246536 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" event={"ID":"3eefce2f-3b96-4b3b-ba40-4d9891bb457e","Type":"ContainerStarted","Data":"09bdcbed994f5a3e1bb8b4ab2035caee5d035a775ea2f03215d68f588965c55c"} Sep 16 00:34:34.246935 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:34.246922 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jzl25" Sep 16 00:34:34.247508 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:34.247488 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jzl25" Sep 16 00:34:34.261730 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:34.261690 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm9zb" podStartSLOduration=3.933544743 podStartE2EDuration="24.261674995s" totalImagesPullingTime="20.328130252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:34:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:34.261184575 +0000 UTC m=+24.623408127" watchObservedRunningTime="2026-09-16 00:34:34.261674995 +0000 UTC m=+24.623898529" Sep 16 00:34:34.326284 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:34.326256 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:34.427916 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:34.427838 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:34.528816 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:34.528737 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:34.872215 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:34.872129 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:34.928915 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:34.928877 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:35.112766 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:35.112730 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:35.112930 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:35.112859 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:36.054737 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:36.054703 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:36.112831 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:36.112804 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:36.113005 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:36.112912 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:36.113005 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:36.112997 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:36.113150 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:36.113102 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:36.152173 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:36.152141 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:36.352497 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:36.352418 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:36.412676 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:36.412646 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:36.469391 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:36.469359 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:36.640977 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:36.640846 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:36.696667 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:36.696640 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:36.799216 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:36.799189 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:36.936690 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:36.936648 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:37.113219 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:37.113195 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:37.113729 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:37.113313 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:37.255270 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:37.255213 2567 generic.go:356] "Generic (PLEG): container finished" podID="8105e12c-26da-4f67-a85b-734fb527ad2b" containerID="ad65e9b9316adbb36de4a6dc013c99e810a05b612d9739644282a82b8a665ba0" exitCode=0 Sep 16 00:34:37.255375 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:37.255309 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9spsl" event={"ID":"8105e12c-26da-4f67-a85b-734fb527ad2b","Type":"ContainerDied","Data":"ad65e9b9316adbb36de4a6dc013c99e810a05b612d9739644282a82b8a665ba0"} Sep 16 00:34:37.259355 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:37.259336 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" event={"ID":"3eefce2f-3b96-4b3b-ba40-4d9891bb457e","Type":"ContainerStarted","Data":"4dcf182b43f378450efc1e5f471b9da446c693ee9f960c1a6c23140c49690e99"} Sep 16 00:34:37.259627 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:37.259605 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:37.259801 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:37.259680 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:37.259801 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:37.259779 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:37.274956 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:37.274935 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:37.275105 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:37.275085 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:34:37.297601 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:37.297565 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" podStartSLOduration=9.110412954 podStartE2EDuration="27.29755201s" totalImagesPullingTime="18.187139056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:37.296806165 +0000 UTC m=+27.659029707" watchObservedRunningTime="2026-09-16 00:34:37.29755201 +0000 UTC m=+27.659775541" Sep 16 00:34:38.112308 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:38.112277 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:38.112563 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:38.112277 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:38.112563 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:38.112401 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:38.112563 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:38.112488 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:38.400565 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:38.400409 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:38.411967 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:38.411952 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cd4mj_abe1a6aa-c150-47db-9b2f-e26f6415570f/dns-node-resolver/0.log" Sep 16 00:34:38.494739 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:38.494697 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:38.817499 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:38.817472 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fpm24"] Sep 16 00:34:38.817630 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:38.817580 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:38.817708 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:38.817686 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:38.819367 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:38.819344 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rxnhx"] Sep 16 00:34:38.819476 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:38.819449 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:38.819576 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:38.819552 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:38.819845 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:38.819827 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gw589"] Sep 16 00:34:38.819927 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:38.819914 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:38.820018 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:38.820002 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:39.254507 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:39.254474 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:39.266004 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:39.265977 2567 generic.go:356] "Generic (PLEG): container finished" podID="8105e12c-26da-4f67-a85b-734fb527ad2b" containerID="c53bc20ab4c1c80d68e5a27f01c478e2967a816643bed93cbeae250945bf52f6" exitCode=0 Sep 16 00:34:39.266155 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:39.266070 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9spsl" event={"ID":"8105e12c-26da-4f67-a85b-734fb527ad2b","Type":"ContainerDied","Data":"c53bc20ab4c1c80d68e5a27f01c478e2967a816643bed93cbeae250945bf52f6"} Sep 16 00:34:39.612633 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:39.612568 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q86vf_d59da895-813d-4240-848d-9513b5886d1c/node-ca/0.log" Sep 16 00:34:40.113904 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:40.113875 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:40.114172 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:40.113987 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:40.395174 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:40.395099 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:40.598234 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:40.598210 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:40.696127 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:40.696096 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:40.747907 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:40.747882 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:41.113060 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:41.113011 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:41.113190 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:41.113134 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:41.113247 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:41.113183 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:41.113295 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:41.113278 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:41.173533 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:41.173506 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:41.272565 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:41.272490 2567 generic.go:356] "Generic (PLEG): container finished" podID="8105e12c-26da-4f67-a85b-734fb527ad2b" containerID="9cf1d4416e5f7768a80a1de49d8cc5bbe19597cadc1b163654bf9e20a9224f2c" exitCode=0 Sep 16 00:34:41.272565 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:41.272540 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9spsl" event={"ID":"8105e12c-26da-4f67-a85b-734fb527ad2b","Type":"ContainerDied","Data":"9cf1d4416e5f7768a80a1de49d8cc5bbe19597cadc1b163654bf9e20a9224f2c"} Sep 16 00:34:41.497278 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:41.497243 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:41.689437 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:41.689370 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:42.014170 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:42.014094 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:42.092091 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:42.092060 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:42.113343 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:42.113313 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:42.113463 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:42.113425 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:42.692572 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:42.692545 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:43.112900 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:43.112871 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:43.113384 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:43.112974 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:43.113384 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:43.113008 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:43.113384 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:43.113101 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:43.284506 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:43.284467 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:43.505902 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:43.505817 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:43.822268 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:43.822241 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret\") pod \"global-pull-secret-syncer-fpm24\" (UID: \"be34ccf1-b7da-47ab-9134-2813a37e07a1\") " pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:43.822446 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:43.822290 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trnvd\" (UniqueName: \"kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd\") pod \"network-check-target-gw589\" (UID: \"f6c0e1cf-272e-412e-b846-d6ee5fc96e28\") " pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:43.822446 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:43.822317 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs\") pod \"network-metrics-daemon-rxnhx\" (UID: \"b08b5ed1-8d6b-402b-88d2-5dbf09029651\") " pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:43.822446 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:43.822395 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 00:34:43.822446 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:43.822398 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 00:34:43.822446 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:43.822411 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 00:34:43.822446 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:43.822430 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 00:34:43.822446 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:43.822442 2567 projected.go:196] Error preparing data for projected volume kube-api-access-trnvd for pod openshift-network-diagnostics/network-check-target-gw589: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 00:34:43.822754 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:43.822459 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret podName:be34ccf1-b7da-47ab-9134-2813a37e07a1 nodeName:}" failed. No retries permitted until 2026-09-16 00:35:15.822438161 +0000 UTC m=+66.184661682 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret") pod "global-pull-secret-syncer-fpm24" (UID: "be34ccf1-b7da-47ab-9134-2813a37e07a1") : object "kube-system"/"original-pull-secret" not registered Sep 16 00:34:43.822754 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:43.822479 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd podName:f6c0e1cf-272e-412e-b846-d6ee5fc96e28 nodeName:}" failed. No retries permitted until 2026-09-16 00:35:15.822467065 +0000 UTC m=+66.184690579 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-trnvd" (UniqueName: "kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd") pod "network-check-target-gw589" (UID: "f6c0e1cf-272e-412e-b846-d6ee5fc96e28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 00:34:43.822754 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:43.822494 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs podName:b08b5ed1-8d6b-402b-88d2-5dbf09029651 nodeName:}" failed. No retries permitted until 2026-09-16 00:35:15.822488805 +0000 UTC m=+66.184712314 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs") pod "network-metrics-daemon-rxnhx" (UID: "b08b5ed1-8d6b-402b-88d2-5dbf09029651") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 00:34:43.894637 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:43.894600 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:44.095775 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:44.095692 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:44.112486 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:44.112462 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:44.112589 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:44.112550 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:44.195924 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:44.195892 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:44.459016 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:44.458931 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:44.519211 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:44.519178 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:45.111247 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:45.111216 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:45.113170 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:45.113146 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:45.113297 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:45.113187 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:45.113348 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:45.113296 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:45.113442 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:45.113422 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:45.205705 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:45.205677 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:45.376194 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:45.376109 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:45.485971 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:45.485932 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:45.689134 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:45.689027 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:46.090154 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:46.089955 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:46.112928 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:46.112906 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:46.113082 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:46.113033 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:46.190067 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:46.190023 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:46.240979 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:46.240955 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:46.286082 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:46.286058 2567 generic.go:356] "Generic (PLEG): container finished" podID="8105e12c-26da-4f67-a85b-734fb527ad2b" containerID="872ed7ddef8015681722ee3a4f5c87243b19a99fd59df66b3bd045cd16eb3fb1" exitCode=0 Sep 16 00:34:46.286163 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:46.286130 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9spsl" event={"ID":"8105e12c-26da-4f67-a85b-734fb527ad2b","Type":"ContainerDied","Data":"872ed7ddef8015681722ee3a4f5c87243b19a99fd59df66b3bd045cd16eb3fb1"} Sep 16 00:34:46.697551 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:46.697481 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:46.773908 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:46.773884 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:47.089322 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:47.089290 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:47.112606 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:47.112585 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:47.112693 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:47.112585 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:47.115377 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:47.112739 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:47.115377 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:47.113174 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:47.287403 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:47.287378 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:47.291200 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:47.291174 2567 generic.go:356] "Generic (PLEG): container finished" podID="8105e12c-26da-4f67-a85b-734fb527ad2b" containerID="423a8b56261d86e24d0ad21c6aeddbfc4d3669c7cb9318bb849431bc9259b1ad" exitCode=0 Sep 16 00:34:47.291321 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:47.291227 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9spsl" event={"ID":"8105e12c-26da-4f67-a85b-734fb527ad2b","Type":"ContainerDied","Data":"423a8b56261d86e24d0ad21c6aeddbfc4d3669c7cb9318bb849431bc9259b1ad"} Sep 16 00:34:47.688133 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:47.688063 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:48.112719 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:48.112693 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:48.112857 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:48.112809 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:48.296573 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:48.296514 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9spsl" event={"ID":"8105e12c-26da-4f67-a85b-734fb527ad2b","Type":"ContainerStarted","Data":"6fed6ddfe8ca34298c9c5d359ac5a99a14e8dc73ccb8204318752465bd14bb82"} Sep 16 00:34:48.318517 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:48.318471 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9spsl" podStartSLOduration=8.761865125 podStartE2EDuration="38.318452908s" totalImagesPullingTime="29.556587783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:34:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:48.316824711 +0000 UTC m=+38.679048242" watchObservedRunningTime="2026-09-16 00:34:48.318452908 +0000 UTC m=+38.680676438" Sep 16 00:34:48.357734 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:48.357705 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:48.411237 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:48.411212 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:48.514050 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:48.514019 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:48.684401 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:48.684377 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:48.887508 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:48.887484 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:49.112404 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:49.112369 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:49.112734 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:49.112467 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:49.112734 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:49.112519 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:49.112734 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:49.112595 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:49.122547 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:49.122524 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:49.283213 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:49.283184 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:49.383755 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:49.383704 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:49.437073 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:49.437028 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:49.946303 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:49.946272 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:50.114013 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:50.113984 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:50.114171 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:50.114076 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:50.296645 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:50.296567 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:50.692184 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:50.692146 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:50.888943 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:50.888901 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:51.087600 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:51.087568 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:51.112229 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:51.112206 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:51.112339 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:51.112206 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:51.112412 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:51.112295 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:51.112460 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:51.112401 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:52.112812 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:52.112784 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:52.113220 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:52.112872 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:53.112730 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:53.112695 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:53.112920 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:53.112695 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:53.112920 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:53.112796 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:53.112920 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:53.112881 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:53.595935 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:53.595888 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:53.697476 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:53.697431 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:53.758699 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:53.758670 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:53.906427 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:53.906368 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:54.113292 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:54.113264 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:54.113655 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:54.113375 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:54.292502 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:54.292418 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:54.343689 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:54.343666 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:54.494900 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:54.494866 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:54.674785 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:54.674759 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:54.771288 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:54.771251 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:55.079597 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:55.079566 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:55.113249 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:55.113226 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:55.113346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:55.113226 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:55.113580 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:55.113342 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:55.113580 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:55.113391 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:55.181898 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:55.181868 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:55.279892 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:55.279857 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:55.336135 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:55.336060 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:56.112997 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:56.112970 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:56.113169 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:56.113085 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw589" podUID="f6c0e1cf-272e-412e-b846-d6ee5fc96e28" Sep 16 00:34:56.303194 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:56.303168 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:56.404244 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:56.404212 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:57.112796 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.112771 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:57.112967 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.112806 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:57.112967 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:57.112878 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpm24" podUID="be34ccf1-b7da-47ab-9134-2813a37e07a1" Sep 16 00:34:57.112967 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:34:57.112930 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rxnhx" podUID="b08b5ed1-8d6b-402b-88d2-5dbf09029651" Sep 16 00:34:57.621888 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.621851 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:57.723497 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.723466 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:57.815849 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.815824 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:57.887960 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.887914 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-194.ec2.internal" event="NodeReady" Sep 16 00:34:57.888065 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.888017 2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 00:34:57.893737 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.893715 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:57.925326 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.925303 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79b88569f4-xrrcb"] Sep 16 00:34:57.948581 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.948562 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:57.951103 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.951074 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bspl6"] Sep 16 00:34:57.952458 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.952435 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sptnt\"" Sep 16 00:34:57.954758 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.952867 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 16 00:34:57.956207 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.956164 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 16 00:34:57.956302 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.956177 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 16 00:34:57.963493 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.963474 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 16 00:34:57.974460 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.974443 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79b88569f4-xrrcb"] Sep 16 00:34:57.974548 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.974471 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w2wk7"] Sep 16 00:34:57.974587 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.974559 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bspl6" Sep 16 00:34:57.978993 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.978973 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 00:34:57.979106 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.979007 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 00:34:57.979167 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.979150 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 00:34:57.979220 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.979192 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cs9kq\"" Sep 16 00:34:57.996918 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.996899 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bspl6"] Sep 16 00:34:57.996999 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.996931 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w2wk7"] Sep 16 00:34:57.996999 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.996993 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w2wk7" Sep 16 00:34:57.999937 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:57.999922 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rpgmh\"" Sep 16 00:34:58.000129 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.000111 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 00:34:58.000129 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.000126 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 00:34:58.043142 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.043125 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jhgj5"] Sep 16 00:34:58.068656 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.068634 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jhgj5"] Sep 16 00:34:58.068733 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.068718 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jhgj5" Sep 16 00:34:58.071584 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.071569 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 00:34:58.071851 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.071831 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 00:34:58.071952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.071921 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q4hl7\"" Sep 16 00:34:58.072056 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.072023 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 00:34:58.072334 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.072319 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 00:34:58.112475 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.112454 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:34:58.117146 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.117130 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wgg7z\"" Sep 16 00:34:58.118000 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.117972 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 00:34:58.118191 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.118177 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 00:34:58.124865 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.124845 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2af5f40b-8530-42d2-b0d6-8c15f74169df-installation-pull-secrets\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.124958 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.124876 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh2w8\" (UniqueName: \"kubernetes.io/projected/2af5f40b-8530-42d2-b0d6-8c15f74169df-kube-api-access-qh2w8\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.124958 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.124904 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6eeff833-d3c0-4fff-b06a-f9870ba630de-metrics-tls\") pod \"dns-default-w2wk7\" (UID: \"6eeff833-d3c0-4fff-b06a-f9870ba630de\") " pod="openshift-dns/dns-default-w2wk7" Sep 16 00:34:58.125064 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.124980 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eeff833-d3c0-4fff-b06a-f9870ba630de-tmp-dir\") pod \"dns-default-w2wk7\" (UID: \"6eeff833-d3c0-4fff-b06a-f9870ba630de\") " pod="openshift-dns/dns-default-w2wk7" Sep 16 00:34:58.125064 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.125004 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2af5f40b-8530-42d2-b0d6-8c15f74169df-ca-trust-extracted\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.125064 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.125024 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qmtm\" (UniqueName: \"kubernetes.io/projected/6eeff833-d3c0-4fff-b06a-f9870ba630de-kube-api-access-9qmtm\") pod \"dns-default-w2wk7\" (UID: \"6eeff833-d3c0-4fff-b06a-f9870ba630de\") " pod="openshift-dns/dns-default-w2wk7" Sep 16 00:34:58.125064 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.125051 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2af5f40b-8530-42d2-b0d6-8c15f74169df-trusted-ca\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.125220 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.125095 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2af5f40b-8530-42d2-b0d6-8c15f74169df-image-registry-private-configuration\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.125220 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.125131 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2af5f40b-8530-42d2-b0d6-8c15f74169df-registry-tls\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.125220 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.125163 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2af5f40b-8530-42d2-b0d6-8c15f74169df-registry-certificates\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.125220 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.125192 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2af5f40b-8530-42d2-b0d6-8c15f74169df-bound-sa-token\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.125220 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.125209 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z42vw\" (UniqueName: \"kubernetes.io/projected/2465e5a4-b67b-42f6-9934-f65c4c2fadb3-kube-api-access-z42vw\") pod \"ingress-canary-bspl6\" (UID: \"2465e5a4-b67b-42f6-9934-f65c4c2fadb3\") " pod="openshift-ingress-canary/ingress-canary-bspl6" Sep 16 00:34:58.125473 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.125231 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6eeff833-d3c0-4fff-b06a-f9870ba630de-config-volume\") pod \"dns-default-w2wk7\" (UID: \"6eeff833-d3c0-4fff-b06a-f9870ba630de\") " pod="openshift-dns/dns-default-w2wk7" Sep 16 00:34:58.125473 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.125249 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2465e5a4-b67b-42f6-9934-f65c4c2fadb3-cert\") pod \"ingress-canary-bspl6\" (UID: \"2465e5a4-b67b-42f6-9934-f65c4c2fadb3\") " pod="openshift-ingress-canary/ingress-canary-bspl6" Sep 16 00:34:58.197109 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.197055 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:58.226143 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.226122 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2af5f40b-8530-42d2-b0d6-8c15f74169df-bound-sa-token\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.226230 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.226148 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z42vw\" (UniqueName: \"kubernetes.io/projected/2465e5a4-b67b-42f6-9934-f65c4c2fadb3-kube-api-access-z42vw\") pod \"ingress-canary-bspl6\" (UID: \"2465e5a4-b67b-42f6-9934-f65c4c2fadb3\") " pod="openshift-ingress-canary/ingress-canary-bspl6" Sep 16 00:34:58.226230 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.226165 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6eeff833-d3c0-4fff-b06a-f9870ba630de-config-volume\") pod \"dns-default-w2wk7\" (UID: \"6eeff833-d3c0-4fff-b06a-f9870ba630de\") " pod="openshift-dns/dns-default-w2wk7" Sep 16 00:34:58.226230 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.226183 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2465e5a4-b67b-42f6-9934-f65c4c2fadb3-cert\") pod \"ingress-canary-bspl6\" (UID: \"2465e5a4-b67b-42f6-9934-f65c4c2fadb3\") " pod="openshift-ingress-canary/ingress-canary-bspl6" Sep 16 00:34:58.226379 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.226310 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2af5f40b-8530-42d2-b0d6-8c15f74169df-installation-pull-secrets\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.226379 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.226344 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh2w8\" (UniqueName: \"kubernetes.io/projected/2af5f40b-8530-42d2-b0d6-8c15f74169df-kube-api-access-qh2w8\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.226379 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.226372 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1596382-2cf4-4b0a-83cd-8e6c2ff16981-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jhgj5\" (UID: \"b1596382-2cf4-4b0a-83cd-8e6c2ff16981\") " pod="openshift-insights/insights-runtime-extractor-jhgj5" Sep 16 00:34:58.226523 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.226403 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6eeff833-d3c0-4fff-b06a-f9870ba630de-metrics-tls\") pod \"dns-default-w2wk7\" (UID: \"6eeff833-d3c0-4fff-b06a-f9870ba630de\") " pod="openshift-dns/dns-default-w2wk7" Sep 16 00:34:58.226523 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.226501 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eeff833-d3c0-4fff-b06a-f9870ba630de-tmp-dir\") pod \"dns-default-w2wk7\" (UID: \"6eeff833-d3c0-4fff-b06a-f9870ba630de\") " pod="openshift-dns/dns-default-w2wk7" Sep 16 00:34:58.226615 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.226534 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2af5f40b-8530-42d2-b0d6-8c15f74169df-ca-trust-extracted\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.226615 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.226580 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qmtm\" (UniqueName: \"kubernetes.io/projected/6eeff833-d3c0-4fff-b06a-f9870ba630de-kube-api-access-9qmtm\") pod \"dns-default-w2wk7\" (UID: \"6eeff833-d3c0-4fff-b06a-f9870ba630de\") " pod="openshift-dns/dns-default-w2wk7" Sep 16 00:34:58.226926 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.226903 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2af5f40b-8530-42d2-b0d6-8c15f74169df-trusted-ca\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.226990 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.226953 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1596382-2cf4-4b0a-83cd-8e6c2ff16981-data-volume\") pod \"insights-runtime-extractor-jhgj5\" (UID: \"b1596382-2cf4-4b0a-83cd-8e6c2ff16981\") " pod="openshift-insights/insights-runtime-extractor-jhgj5" Sep 16 00:34:58.227083 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.226995 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2af5f40b-8530-42d2-b0d6-8c15f74169df-image-registry-private-configuration\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.227083 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.227066 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1596382-2cf4-4b0a-83cd-8e6c2ff16981-crio-socket\") pod \"insights-runtime-extractor-jhgj5\" (UID: \"b1596382-2cf4-4b0a-83cd-8e6c2ff16981\") " pod="openshift-insights/insights-runtime-extractor-jhgj5" Sep 16 00:34:58.227191 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.227101 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1596382-2cf4-4b0a-83cd-8e6c2ff16981-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jhgj5\" (UID: \"b1596382-2cf4-4b0a-83cd-8e6c2ff16981\") " pod="openshift-insights/insights-runtime-extractor-jhgj5" Sep 16 00:34:58.227191 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.227133 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhgdc\" (UniqueName: \"kubernetes.io/projected/b1596382-2cf4-4b0a-83cd-8e6c2ff16981-kube-api-access-rhgdc\") pod \"insights-runtime-extractor-jhgj5\" (UID: \"b1596382-2cf4-4b0a-83cd-8e6c2ff16981\") " pod="openshift-insights/insights-runtime-extractor-jhgj5" Sep 16 00:34:58.227191 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.227165 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2af5f40b-8530-42d2-b0d6-8c15f74169df-registry-tls\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.227340 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.227205 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2af5f40b-8530-42d2-b0d6-8c15f74169df-registry-certificates\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.227340 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.227217 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eeff833-d3c0-4fff-b06a-f9870ba630de-tmp-dir\") pod \"dns-default-w2wk7\" (UID: \"6eeff833-d3c0-4fff-b06a-f9870ba630de\") " pod="openshift-dns/dns-default-w2wk7" Sep 16 00:34:58.227444 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.227365 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2af5f40b-8530-42d2-b0d6-8c15f74169df-ca-trust-extracted\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.228087 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.228066 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2af5f40b-8530-42d2-b0d6-8c15f74169df-registry-certificates\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.228229 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.228201 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6eeff833-d3c0-4fff-b06a-f9870ba630de-config-volume\") pod \"dns-default-w2wk7\" (UID: \"6eeff833-d3c0-4fff-b06a-f9870ba630de\") " pod="openshift-dns/dns-default-w2wk7" Sep 16 00:34:58.230374 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.230348 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2af5f40b-8530-42d2-b0d6-8c15f74169df-trusted-ca\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.232269 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.232248 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2465e5a4-b67b-42f6-9934-f65c4c2fadb3-cert\") pod \"ingress-canary-bspl6\" (UID: \"2465e5a4-b67b-42f6-9934-f65c4c2fadb3\") " pod="openshift-ingress-canary/ingress-canary-bspl6" Sep 16 00:34:58.232399 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.232254 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2af5f40b-8530-42d2-b0d6-8c15f74169df-registry-tls\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.233339 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.233311 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2af5f40b-8530-42d2-b0d6-8c15f74169df-image-registry-private-configuration\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.233769 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.233703 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6eeff833-d3c0-4fff-b06a-f9870ba630de-metrics-tls\") pod \"dns-default-w2wk7\" (UID: \"6eeff833-d3c0-4fff-b06a-f9870ba630de\") " pod="openshift-dns/dns-default-w2wk7" Sep 16 00:34:58.234564 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.234539 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2af5f40b-8530-42d2-b0d6-8c15f74169df-installation-pull-secrets\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.234943 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.234925 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh2w8\" (UniqueName: \"kubernetes.io/projected/2af5f40b-8530-42d2-b0d6-8c15f74169df-kube-api-access-qh2w8\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.235291 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.235275 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2af5f40b-8530-42d2-b0d6-8c15f74169df-bound-sa-token\") pod \"image-registry-79b88569f4-xrrcb\" (UID: \"2af5f40b-8530-42d2-b0d6-8c15f74169df\") " pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.235629 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.235601 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qmtm\" (UniqueName: \"kubernetes.io/projected/6eeff833-d3c0-4fff-b06a-f9870ba630de-kube-api-access-9qmtm\") pod \"dns-default-w2wk7\" (UID: \"6eeff833-d3c0-4fff-b06a-f9870ba630de\") " pod="openshift-dns/dns-default-w2wk7" Sep 16 00:34:58.236143 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.236125 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z42vw\" (UniqueName: \"kubernetes.io/projected/2465e5a4-b67b-42f6-9934-f65c4c2fadb3-kube-api-access-z42vw\") pod \"ingress-canary-bspl6\" (UID: \"2465e5a4-b67b-42f6-9934-f65c4c2fadb3\") " pod="openshift-ingress-canary/ingress-canary-bspl6" Sep 16 00:34:58.259976 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.259953 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:58.261751 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.261731 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:58.283085 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.283054 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bspl6" Sep 16 00:34:58.303990 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.303972 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w2wk7" Sep 16 00:34:58.327954 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.327926 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1596382-2cf4-4b0a-83cd-8e6c2ff16981-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jhgj5\" (UID: \"b1596382-2cf4-4b0a-83cd-8e6c2ff16981\") " pod="openshift-insights/insights-runtime-extractor-jhgj5" Sep 16 00:34:58.328085 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.327987 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1596382-2cf4-4b0a-83cd-8e6c2ff16981-data-volume\") pod \"insights-runtime-extractor-jhgj5\" (UID: \"b1596382-2cf4-4b0a-83cd-8e6c2ff16981\") " pod="openshift-insights/insights-runtime-extractor-jhgj5" Sep 16 00:34:58.328085 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.328015 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1596382-2cf4-4b0a-83cd-8e6c2ff16981-crio-socket\") pod \"insights-runtime-extractor-jhgj5\" (UID: \"b1596382-2cf4-4b0a-83cd-8e6c2ff16981\") " pod="openshift-insights/insights-runtime-extractor-jhgj5" Sep 16 00:34:58.328085 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.328057 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1596382-2cf4-4b0a-83cd-8e6c2ff16981-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jhgj5\" (UID: \"b1596382-2cf4-4b0a-83cd-8e6c2ff16981\") " pod="openshift-insights/insights-runtime-extractor-jhgj5" Sep 16 00:34:58.328251 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.328115 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhgdc\" (UniqueName: \"kubernetes.io/projected/b1596382-2cf4-4b0a-83cd-8e6c2ff16981-kube-api-access-rhgdc\") pod \"insights-runtime-extractor-jhgj5\" (UID: \"b1596382-2cf4-4b0a-83cd-8e6c2ff16981\") " pod="openshift-insights/insights-runtime-extractor-jhgj5" Sep 16 00:34:58.328546 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.328324 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1596382-2cf4-4b0a-83cd-8e6c2ff16981-data-volume\") pod \"insights-runtime-extractor-jhgj5\" (UID: \"b1596382-2cf4-4b0a-83cd-8e6c2ff16981\") " pod="openshift-insights/insights-runtime-extractor-jhgj5" Sep 16 00:34:58.328546 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.328401 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1596382-2cf4-4b0a-83cd-8e6c2ff16981-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jhgj5\" (UID: \"b1596382-2cf4-4b0a-83cd-8e6c2ff16981\") " pod="openshift-insights/insights-runtime-extractor-jhgj5" Sep 16 00:34:58.328546 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.328498 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1596382-2cf4-4b0a-83cd-8e6c2ff16981-crio-socket\") pod \"insights-runtime-extractor-jhgj5\" (UID: \"b1596382-2cf4-4b0a-83cd-8e6c2ff16981\") " pod="openshift-insights/insights-runtime-extractor-jhgj5" Sep 16 00:34:58.330720 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.330697 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1596382-2cf4-4b0a-83cd-8e6c2ff16981-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jhgj5\" (UID: \"b1596382-2cf4-4b0a-83cd-8e6c2ff16981\") " pod="openshift-insights/insights-runtime-extractor-jhgj5" Sep 16 00:34:58.336383 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.336238 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhgdc\" (UniqueName: \"kubernetes.io/projected/b1596382-2cf4-4b0a-83cd-8e6c2ff16981-kube-api-access-rhgdc\") pod \"insights-runtime-extractor-jhgj5\" (UID: \"b1596382-2cf4-4b0a-83cd-8e6c2ff16981\") " pod="openshift-insights/insights-runtime-extractor-jhgj5" Sep 16 00:34:58.378663 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.376990 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jhgj5" Sep 16 00:34:58.558358 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.558333 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:58.598721 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.598701 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w2wk7"] Sep 16 00:34:58.601579 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:58.601545 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eeff833_d3c0_4fff_b06a_f9870ba630de.slice/crio-4d2e2d6a8502ede934adcd881b0d041dacfc37cbf3929747d112c6979e5ea41e WatchSource:0}: Error finding container 4d2e2d6a8502ede934adcd881b0d041dacfc37cbf3929747d112c6979e5ea41e: Status 404 returned error can't find the container with id 4d2e2d6a8502ede934adcd881b0d041dacfc37cbf3929747d112c6979e5ea41e Sep 16 00:34:58.601708 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.601590 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bspl6"] Sep 16 00:34:58.604401 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.604379 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79b88569f4-xrrcb"] Sep 16 00:34:58.604504 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:58.604470 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2465e5a4_b67b_42f6_9934_f65c4c2fadb3.slice/crio-6c393323d65826bd3433ca607af142e0a78ab4b1323153347cb22f88bbacf818 WatchSource:0}: Error finding container 6c393323d65826bd3433ca607af142e0a78ab4b1323153347cb22f88bbacf818: Status 404 returned error can't find the container with id 6c393323d65826bd3433ca607af142e0a78ab4b1323153347cb22f88bbacf818 Sep 16 00:34:58.607177 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:58.607160 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af5f40b_8530_42d2_b0d6_8c15f74169df.slice/crio-b6ebac47fd85e6b0a1604bfeb40ed615852745437ec5f9af05b44942e137be5c WatchSource:0}: Error finding container b6ebac47fd85e6b0a1604bfeb40ed615852745437ec5f9af05b44942e137be5c: Status 404 returned error can't find the container with id b6ebac47fd85e6b0a1604bfeb40ed615852745437ec5f9af05b44942e137be5c Sep 16 00:34:58.671304 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.671281 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jhgj5"] Sep 16 00:34:58.673805 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:34:58.673782 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1596382_2cf4_4b0a_83cd_8e6c2ff16981.slice/crio-c4ea12bd721e52b4a2290cedb51cd9474335da3880fbedd6146c634166f86742 WatchSource:0}: Error finding container c4ea12bd721e52b4a2290cedb51cd9474335da3880fbedd6146c634166f86742: Status 404 returned error can't find the container with id c4ea12bd721e52b4a2290cedb51cd9474335da3880fbedd6146c634166f86742 Sep 16 00:34:58.947178 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:58.947129 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:59.002009 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:59.001935 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:59.113024 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:59.112994 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:34:59.113231 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:59.113209 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:34:59.115919 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:59.115896 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g7jvk\"" Sep 16 00:34:59.116064 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:59.115941 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 00:34:59.116236 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:59.116219 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 00:34:59.323500 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:59.323010 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" event={"ID":"2af5f40b-8530-42d2-b0d6-8c15f74169df","Type":"ContainerStarted","Data":"f6b8905a2dc624c9ca9dee538d030084516391b84f54e23776a8080d55a79fe9"} Sep 16 00:34:59.323500 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:59.323090 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" event={"ID":"2af5f40b-8530-42d2-b0d6-8c15f74169df","Type":"ContainerStarted","Data":"b6ebac47fd85e6b0a1604bfeb40ed615852745437ec5f9af05b44942e137be5c"} Sep 16 00:34:59.323500 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:59.323307 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:34:59.324922 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:59.324896 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w2wk7" event={"ID":"6eeff833-d3c0-4fff-b06a-f9870ba630de","Type":"ContainerStarted","Data":"4d2e2d6a8502ede934adcd881b0d041dacfc37cbf3929747d112c6979e5ea41e"} Sep 16 00:34:59.326419 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:59.326385 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bspl6" event={"ID":"2465e5a4-b67b-42f6-9934-f65c4c2fadb3","Type":"ContainerStarted","Data":"6c393323d65826bd3433ca607af142e0a78ab4b1323153347cb22f88bbacf818"} Sep 16 00:34:59.328092 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:59.328068 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jhgj5" event={"ID":"b1596382-2cf4-4b0a-83cd-8e6c2ff16981","Type":"ContainerStarted","Data":"a2c2a2c8a99d005bad939d09c305b73238b1ccfc5738eae5b382a0772435de1d"} Sep 16 00:34:59.328202 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:59.328122 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jhgj5" event={"ID":"b1596382-2cf4-4b0a-83cd-8e6c2ff16981","Type":"ContainerStarted","Data":"c4ea12bd721e52b4a2290cedb51cd9474335da3880fbedd6146c634166f86742"} Sep 16 00:34:59.509024 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:59.508990 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:34:59.578239 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:34:59.578151 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:00.332596 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:00.332554 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jhgj5" event={"ID":"b1596382-2cf4-4b0a-83cd-8e6c2ff16981","Type":"ContainerStarted","Data":"1f684a476963a84f96202147ec574783523207256394adcb2914a527d76580f1"} Sep 16 00:35:01.507658 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:01.507628 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:01.614364 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:01.614333 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:03.342135 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:03.342094 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w2wk7" event={"ID":"6eeff833-d3c0-4fff-b06a-f9870ba630de","Type":"ContainerStarted","Data":"33501cc0f8adf9579697e622f61235a25c6d4f10f90cf9e5095b1587da43b634"} Sep 16 00:35:03.342605 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:03.342183 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w2wk7" event={"ID":"6eeff833-d3c0-4fff-b06a-f9870ba630de","Type":"ContainerStarted","Data":"79acbdacbaa3d7c924e79419125b96359d7e23ded8ed530f8f2c595bb610293b"} Sep 16 00:35:03.342605 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:03.342236 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w2wk7" Sep 16 00:35:03.343661 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:03.343642 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bspl6" event={"ID":"2465e5a4-b67b-42f6-9934-f65c4c2fadb3","Type":"ContainerStarted","Data":"a903b647c56cc36778c673df72cfef4d55885cb6c3b2d3c3f143f2b66596e911"} Sep 16 00:35:03.345477 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:03.345455 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jhgj5" event={"ID":"b1596382-2cf4-4b0a-83cd-8e6c2ff16981","Type":"ContainerStarted","Data":"e2bb41785346175b8fbe40e78e32b7b1804380c41e417cde60d3320a01902022"} Sep 16 00:35:03.359203 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:03.359166 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" podStartSLOduration=6.359154315 podStartE2EDuration="6.359154315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:59.342225865 +0000 UTC m=+49.704449411" watchObservedRunningTime="2026-09-16 00:35:03.359154315 +0000 UTC m=+53.721377847" Sep 16 00:35:03.359319 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:03.359294 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w2wk7" podStartSLOduration=2.7261850450000003 podStartE2EDuration="6.359285118s" totalImagesPullingTime="3.633100073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:03.357907972 +0000 UTC m=+53.720131524" watchObservedRunningTime="2026-09-16 00:35:03.359285118 +0000 UTC m=+53.721508650" Sep 16 00:35:03.376641 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:03.376599 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bspl6" podStartSLOduration=2.739063844 podStartE2EDuration="6.376586905s" totalImagesPullingTime="3.637523061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:03.376192413 +0000 UTC m=+53.738415962" watchObservedRunningTime="2026-09-16 00:35:03.376586905 +0000 UTC m=+53.738810437" Sep 16 00:35:03.392833 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:03.392798 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jhgj5" podStartSLOduration=1.6258542679999999 podStartE2EDuration="5.392785866s" totalImagesPullingTime="3.766931598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:34:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:03.392188006 +0000 UTC m=+53.754411556" watchObservedRunningTime="2026-09-16 00:35:03.392785866 +0000 UTC m=+53.755009397" Sep 16 00:35:04.382172 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:04.382140 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:04.432175 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:04.432146 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:08.911772 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:08.911740 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:09.013088 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:09.013051 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:09.113480 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:09.113457 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:09.284264 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:09.284211 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-75ccc" Sep 16 00:35:09.409903 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:09.409880 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:09.510747 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:09.510720 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:09.961702 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:09.961671 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:10.031055 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:10.031007 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:10.155878 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:10.155845 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:10.214869 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:10.214803 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:10.334871 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:10.334834 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:10.566547 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:10.566517 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:10.622283 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:10.622255 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:12.107872 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:12.107829 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:12.159150 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:12.159121 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:13.206149 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.206115 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:13.246853 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.246814 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-76pzb"] Sep 16 00:35:13.249772 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.249752 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-76pzb" Sep 16 00:35:13.252110 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.252088 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-76pzb"] Sep 16 00:35:13.252716 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.252668 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 00:35:13.252921 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.252895 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 00:35:13.253060 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.252948 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 16 00:35:13.253060 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.252966 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 16 00:35:13.254170 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.254147 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 00:35:13.254170 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.254165 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fl5rh\"" Sep 16 00:35:13.271270 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.271249 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:13.351644 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.351628 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w2wk7" Sep 16 00:35:13.442739 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.442714 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/687690d1-dc38-41a7-88a5-ebbb24b3f30d-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-76pzb\" (UID: \"687690d1-dc38-41a7-88a5-ebbb24b3f30d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-76pzb" Sep 16 00:35:13.442874 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.442765 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c848j\" (UniqueName: \"kubernetes.io/projected/687690d1-dc38-41a7-88a5-ebbb24b3f30d-kube-api-access-c848j\") pod \"prometheus-operator-68bcd7f987-76pzb\" (UID: \"687690d1-dc38-41a7-88a5-ebbb24b3f30d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-76pzb" Sep 16 00:35:13.442874 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.442787 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/687690d1-dc38-41a7-88a5-ebbb24b3f30d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-76pzb\" (UID: \"687690d1-dc38-41a7-88a5-ebbb24b3f30d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-76pzb" Sep 16 00:35:13.442948 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.442863 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/687690d1-dc38-41a7-88a5-ebbb24b3f30d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-76pzb\" (UID: \"687690d1-dc38-41a7-88a5-ebbb24b3f30d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-76pzb" Sep 16 00:35:13.543887 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.543835 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/687690d1-dc38-41a7-88a5-ebbb24b3f30d-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-76pzb\" (UID: \"687690d1-dc38-41a7-88a5-ebbb24b3f30d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-76pzb" Sep 16 00:35:13.543964 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.543888 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c848j\" (UniqueName: \"kubernetes.io/projected/687690d1-dc38-41a7-88a5-ebbb24b3f30d-kube-api-access-c848j\") pod \"prometheus-operator-68bcd7f987-76pzb\" (UID: \"687690d1-dc38-41a7-88a5-ebbb24b3f30d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-76pzb" Sep 16 00:35:13.544098 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.544078 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/687690d1-dc38-41a7-88a5-ebbb24b3f30d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-76pzb\" (UID: \"687690d1-dc38-41a7-88a5-ebbb24b3f30d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-76pzb" Sep 16 00:35:13.544182 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:35:13.544166 2567 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 16 00:35:13.544221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.544209 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/687690d1-dc38-41a7-88a5-ebbb24b3f30d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-76pzb\" (UID: \"687690d1-dc38-41a7-88a5-ebbb24b3f30d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-76pzb" Sep 16 00:35:13.544254 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:35:13.544231 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/687690d1-dc38-41a7-88a5-ebbb24b3f30d-prometheus-operator-tls podName:687690d1-dc38-41a7-88a5-ebbb24b3f30d nodeName:}" failed. No retries permitted until 2026-09-16 00:35:14.044212396 +0000 UTC m=+64.406435911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/687690d1-dc38-41a7-88a5-ebbb24b3f30d-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-76pzb" (UID: "687690d1-dc38-41a7-88a5-ebbb24b3f30d") : secret "prometheus-operator-tls" not found Sep 16 00:35:13.544556 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.544538 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/687690d1-dc38-41a7-88a5-ebbb24b3f30d-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-76pzb\" (UID: \"687690d1-dc38-41a7-88a5-ebbb24b3f30d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-76pzb" Sep 16 00:35:13.546504 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.546485 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/687690d1-dc38-41a7-88a5-ebbb24b3f30d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-76pzb\" (UID: \"687690d1-dc38-41a7-88a5-ebbb24b3f30d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-76pzb" Sep 16 00:35:13.553188 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.553168 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c848j\" (UniqueName: \"kubernetes.io/projected/687690d1-dc38-41a7-88a5-ebbb24b3f30d-kube-api-access-c848j\") pod \"prometheus-operator-68bcd7f987-76pzb\" (UID: \"687690d1-dc38-41a7-88a5-ebbb24b3f30d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-76pzb" Sep 16 00:35:13.711681 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:13.711659 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:14.047663 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:14.047632 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/687690d1-dc38-41a7-88a5-ebbb24b3f30d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-76pzb\" (UID: \"687690d1-dc38-41a7-88a5-ebbb24b3f30d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-76pzb" Sep 16 00:35:14.050092 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:14.050062 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/687690d1-dc38-41a7-88a5-ebbb24b3f30d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-76pzb\" (UID: \"687690d1-dc38-41a7-88a5-ebbb24b3f30d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-76pzb" Sep 16 00:35:14.159798 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:14.159777 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-76pzb" Sep 16 00:35:14.428844 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:14.428817 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-76pzb"] Sep 16 00:35:14.431901 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:35:14.431868 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687690d1_dc38_41a7_88a5_ebbb24b3f30d.slice/crio-8704f496442d1a8c34f0e4fd1622b365a922c1d8c96ef8e1b0737f8e0beadbc1 WatchSource:0}: Error finding container 8704f496442d1a8c34f0e4fd1622b365a922c1d8c96ef8e1b0737f8e0beadbc1: Status 404 returned error can't find the container with id 8704f496442d1a8c34f0e4fd1622b365a922c1d8c96ef8e1b0737f8e0beadbc1 Sep 16 00:35:15.384861 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:15.384832 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-76pzb" event={"ID":"687690d1-dc38-41a7-88a5-ebbb24b3f30d","Type":"ContainerStarted","Data":"8704f496442d1a8c34f0e4fd1622b365a922c1d8c96ef8e1b0737f8e0beadbc1"} Sep 16 00:35:15.861259 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:15.861231 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret\") pod \"global-pull-secret-syncer-fpm24\" (UID: \"be34ccf1-b7da-47ab-9134-2813a37e07a1\") " pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:35:15.861641 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:15.861275 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trnvd\" (UniqueName: \"kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd\") pod \"network-check-target-gw589\" (UID: \"f6c0e1cf-272e-412e-b846-d6ee5fc96e28\") " pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:35:15.861641 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:15.861295 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs\") pod \"network-metrics-daemon-rxnhx\" (UID: \"b08b5ed1-8d6b-402b-88d2-5dbf09029651\") " pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:35:15.863960 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:15.863942 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 00:35:15.865144 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:15.865126 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 00:35:15.865230 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:15.865128 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 00:35:15.874739 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:15.874717 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be34ccf1-b7da-47ab-9134-2813a37e07a1-original-pull-secret\") pod \"global-pull-secret-syncer-fpm24\" (UID: \"be34ccf1-b7da-47ab-9134-2813a37e07a1\") " pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:35:15.874819 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:15.874754 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b08b5ed1-8d6b-402b-88d2-5dbf09029651-metrics-certs\") pod \"network-metrics-daemon-rxnhx\" (UID: \"b08b5ed1-8d6b-402b-88d2-5dbf09029651\") " pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:35:15.874819 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:15.874812 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 00:35:15.885176 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:15.885153 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trnvd\" (UniqueName: \"kubernetes.io/projected/f6c0e1cf-272e-412e-b846-d6ee5fc96e28-kube-api-access-trnvd\") pod \"network-check-target-gw589\" (UID: \"f6c0e1cf-272e-412e-b846-d6ee5fc96e28\") " pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:35:15.926016 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:15.926000 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpm24" Sep 16 00:35:15.934354 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:15.934336 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g7jvk\"" Sep 16 00:35:15.942775 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:15.942757 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rxnhx" Sep 16 00:35:16.122439 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:16.122368 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wgg7z\"" Sep 16 00:35:16.130713 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:16.130671 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:35:16.207634 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:16.207608 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fpm24"] Sep 16 00:35:16.212189 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:35:16.211753 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe34ccf1_b7da_47ab_9134_2813a37e07a1.slice/crio-4335f2f3d5ab8e76c0d9f0a089fe50b6bb9cecd2dab1fc35239fadf54ecdedcf WatchSource:0}: Error finding container 4335f2f3d5ab8e76c0d9f0a089fe50b6bb9cecd2dab1fc35239fadf54ecdedcf: Status 404 returned error can't find the container with id 4335f2f3d5ab8e76c0d9f0a089fe50b6bb9cecd2dab1fc35239fadf54ecdedcf Sep 16 00:35:16.234793 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:16.234764 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rxnhx"] Sep 16 00:35:16.238321 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:35:16.238298 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08b5ed1_8d6b_402b_88d2_5dbf09029651.slice/crio-c487900fd64ef1600632c4499610a432731a4ff57113c6974e2c5a50b96bad2f WatchSource:0}: Error finding container c487900fd64ef1600632c4499610a432731a4ff57113c6974e2c5a50b96bad2f: Status 404 returned error can't find the container with id c487900fd64ef1600632c4499610a432731a4ff57113c6974e2c5a50b96bad2f Sep 16 00:35:16.270561 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:16.270535 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:16.388536 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:16.388470 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fpm24" event={"ID":"be34ccf1-b7da-47ab-9134-2813a37e07a1","Type":"ContainerStarted","Data":"4335f2f3d5ab8e76c0d9f0a089fe50b6bb9cecd2dab1fc35239fadf54ecdedcf"} Sep 16 00:35:16.390422 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:16.390391 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-76pzb" event={"ID":"687690d1-dc38-41a7-88a5-ebbb24b3f30d","Type":"ContainerStarted","Data":"8600e5b337d622042e238ae795da1b387d19f161ff7796e6f84edd8c06e71857"} Sep 16 00:35:16.390513 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:16.390441 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-76pzb" event={"ID":"687690d1-dc38-41a7-88a5-ebbb24b3f30d","Type":"ContainerStarted","Data":"114883781bee89a85ffc8da25d85a93f00d5305cd81561e9cf9062a510aed489"} Sep 16 00:35:16.391582 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:16.391563 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rxnhx" event={"ID":"b08b5ed1-8d6b-402b-88d2-5dbf09029651","Type":"ContainerStarted","Data":"c487900fd64ef1600632c4499610a432731a4ff57113c6974e2c5a50b96bad2f"} Sep 16 00:35:16.405852 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:16.405804 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-76pzb" podStartSLOduration=1.9323993430000002 podStartE2EDuration="3.405786964s" totalImagesPullingTime="1.473387621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:16.405322266 +0000 UTC m=+66.767545819" watchObservedRunningTime="2026-09-16 00:35:16.405786964 +0000 UTC m=+66.768010497" Sep 16 00:35:16.424194 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:16.424175 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gw589"] Sep 16 00:35:16.426836 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:35:16.426817 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c0e1cf_272e_412e_b846_d6ee5fc96e28.slice/crio-7ad893890eb2df063f8601d53a8083722a25c22cf7bdacb2bc9ff55b0c828d3c WatchSource:0}: Error finding container 7ad893890eb2df063f8601d53a8083722a25c22cf7bdacb2bc9ff55b0c828d3c: Status 404 returned error can't find the container with id 7ad893890eb2df063f8601d53a8083722a25c22cf7bdacb2bc9ff55b0c828d3c Sep 16 00:35:17.149241 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:17.149196 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:17.207455 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:17.207425 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:17.311972 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:17.311940 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:17.366726 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:17.366695 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:17.397837 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:17.397800 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gw589" event={"ID":"f6c0e1cf-272e-412e-b846-d6ee5fc96e28","Type":"ContainerStarted","Data":"7ad893890eb2df063f8601d53a8083722a25c22cf7bdacb2bc9ff55b0c828d3c"} Sep 16 00:35:17.714372 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:17.714288 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:17.770214 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:17.770183 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:18.111201 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.111171 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:18.165808 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.165766 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:18.266425 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.266382 2567 patch_prober.go:28] interesting pod/image-registry-79b88569f4-xrrcb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 16 00:35:18.266594 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.266446 2567 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" podUID="2af5f40b-8530-42d2-b0d6-8c15f74169df" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 00:35:18.348391 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.348353 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:18.404673 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.404592 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rxnhx" event={"ID":"b08b5ed1-8d6b-402b-88d2-5dbf09029651","Type":"ContainerStarted","Data":"594574ba574755e5af0f5a1ea7b608d630860ee54ab7a7b2b5732cae8605d244"} Sep 16 00:35:18.404673 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.404657 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rxnhx" event={"ID":"b08b5ed1-8d6b-402b-88d2-5dbf09029651","Type":"ContainerStarted","Data":"02a68f7dcfeeb581c5d1f0740f3d50d2797071ba13b8bad895878798808bb33e"} Sep 16 00:35:18.422022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.421977 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rxnhx" podStartSLOduration=67.280757803 podStartE2EDuration="1m8.421962244s" totalImagesPullingTime="1.141204441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:18.419891047 +0000 UTC m=+68.782114626" watchObservedRunningTime="2026-09-16 00:35:18.421962244 +0000 UTC m=+68.784185775" Sep 16 00:35:18.511001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.510886 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:18.581603 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.581574 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz"] Sep 16 00:35:18.583926 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.583902 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" Sep 16 00:35:18.586414 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.586392 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 16 00:35:18.586534 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.586446 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kkgm2\"" Sep 16 00:35:18.586534 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.586445 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 16 00:35:18.597942 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.596397 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz"] Sep 16 00:35:18.603133 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.603111 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h8w5x"] Sep 16 00:35:18.606052 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.606009 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.606291 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.606213 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:18.609877 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.609857 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 00:35:18.609877 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.609872 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 00:35:18.610071 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.609921 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f98b9\"" Sep 16 00:35:18.610071 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.609862 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 00:35:18.610176 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.610081 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9"] Sep 16 00:35:18.612907 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.612886 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" Sep 16 00:35:18.615709 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.615691 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 16 00:35:18.615937 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.615916 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-cthpr\"" Sep 16 00:35:18.616154 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.616139 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 16 00:35:18.616421 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.616406 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 16 00:35:18.625558 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.625537 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9"] Sep 16 00:35:18.685218 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.685147 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca98b25c-bcff-40b9-8182-15f72dde4b26-node-exporter-accelerators-collector-config\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.685218 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.685197 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d4219f4-4ffc-4733-b732-eb264c641245-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-sldj9\" (UID: \"2d4219f4-4ffc-4733-b732-eb264c641245\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" Sep 16 00:35:18.685378 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.685231 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8746e8b6-9c83-440d-a0f9-4b9070ef6104-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-xsrqz\" (UID: \"8746e8b6-9c83-440d-a0f9-4b9070ef6104\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" Sep 16 00:35:18.685378 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.685274 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d4219f4-4ffc-4733-b732-eb264c641245-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-sldj9\" (UID: \"2d4219f4-4ffc-4733-b732-eb264c641245\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" Sep 16 00:35:18.685378 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.685313 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca98b25c-bcff-40b9-8182-15f72dde4b26-node-exporter-textfile\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.685378 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.685355 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca98b25c-bcff-40b9-8182-15f72dde4b26-metrics-client-ca\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.685546 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.685386 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca98b25c-bcff-40b9-8182-15f72dde4b26-sys\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.685546 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.685401 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdvfb\" (UniqueName: \"kubernetes.io/projected/8746e8b6-9c83-440d-a0f9-4b9070ef6104-kube-api-access-zdvfb\") pod \"openshift-state-metrics-67dcf8c979-xsrqz\" (UID: \"8746e8b6-9c83-440d-a0f9-4b9070ef6104\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" Sep 16 00:35:18.685546 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.685428 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2d4219f4-4ffc-4733-b732-eb264c641245-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-sldj9\" (UID: \"2d4219f4-4ffc-4733-b732-eb264c641245\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" Sep 16 00:35:18.685546 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.685506 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca98b25c-bcff-40b9-8182-15f72dde4b26-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.685692 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.685546 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2d4219f4-4ffc-4733-b732-eb264c641245-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-sldj9\" (UID: \"2d4219f4-4ffc-4733-b732-eb264c641245\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" Sep 16 00:35:18.685692 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.685573 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8746e8b6-9c83-440d-a0f9-4b9070ef6104-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-xsrqz\" (UID: \"8746e8b6-9c83-440d-a0f9-4b9070ef6104\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" Sep 16 00:35:18.685692 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.685601 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9z69\" (UniqueName: \"kubernetes.io/projected/ca98b25c-bcff-40b9-8182-15f72dde4b26-kube-api-access-j9z69\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.685692 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.685636 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca98b25c-bcff-40b9-8182-15f72dde4b26-node-exporter-wtmp\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.685692 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.685662 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4219f4-4ffc-4733-b732-eb264c641245-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-sldj9\" (UID: \"2d4219f4-4ffc-4733-b732-eb264c641245\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" Sep 16 00:35:18.685870 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.685697 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z68z\" (UniqueName: \"kubernetes.io/projected/2d4219f4-4ffc-4733-b732-eb264c641245-kube-api-access-9z68z\") pod \"kube-state-metrics-59fbdd77d-sldj9\" (UID: \"2d4219f4-4ffc-4733-b732-eb264c641245\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" Sep 16 00:35:18.685870 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.685717 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca98b25c-bcff-40b9-8182-15f72dde4b26-node-exporter-tls\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.685870 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.685743 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca98b25c-bcff-40b9-8182-15f72dde4b26-root\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.685870 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.685768 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8746e8b6-9c83-440d-a0f9-4b9070ef6104-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-xsrqz\" (UID: \"8746e8b6-9c83-440d-a0f9-4b9070ef6104\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" Sep 16 00:35:18.711164 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.711140 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:18.786784 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.786746 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca98b25c-bcff-40b9-8182-15f72dde4b26-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.786924 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.786797 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2d4219f4-4ffc-4733-b732-eb264c641245-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-sldj9\" (UID: \"2d4219f4-4ffc-4733-b732-eb264c641245\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" Sep 16 00:35:18.786924 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.786841 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8746e8b6-9c83-440d-a0f9-4b9070ef6104-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-xsrqz\" (UID: \"8746e8b6-9c83-440d-a0f9-4b9070ef6104\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" Sep 16 00:35:18.786924 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.786867 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9z69\" (UniqueName: \"kubernetes.io/projected/ca98b25c-bcff-40b9-8182-15f72dde4b26-kube-api-access-j9z69\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.786924 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.786897 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca98b25c-bcff-40b9-8182-15f72dde4b26-node-exporter-wtmp\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.787168 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.787029 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca98b25c-bcff-40b9-8182-15f72dde4b26-node-exporter-wtmp\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.787168 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.787093 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4219f4-4ffc-4733-b732-eb264c641245-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-sldj9\" (UID: \"2d4219f4-4ffc-4733-b732-eb264c641245\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" Sep 16 00:35:18.787168 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.787134 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z68z\" (UniqueName: \"kubernetes.io/projected/2d4219f4-4ffc-4733-b732-eb264c641245-kube-api-access-9z68z\") pod \"kube-state-metrics-59fbdd77d-sldj9\" (UID: \"2d4219f4-4ffc-4733-b732-eb264c641245\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" Sep 16 00:35:18.787168 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.787161 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca98b25c-bcff-40b9-8182-15f72dde4b26-node-exporter-tls\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.787361 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.787188 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca98b25c-bcff-40b9-8182-15f72dde4b26-root\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.787361 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.787247 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2d4219f4-4ffc-4733-b732-eb264c641245-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-sldj9\" (UID: \"2d4219f4-4ffc-4733-b732-eb264c641245\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" Sep 16 00:35:18.787361 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.787267 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca98b25c-bcff-40b9-8182-15f72dde4b26-root\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.787501 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.787448 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8746e8b6-9c83-440d-a0f9-4b9070ef6104-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-xsrqz\" (UID: \"8746e8b6-9c83-440d-a0f9-4b9070ef6104\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" Sep 16 00:35:18.787501 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.787485 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca98b25c-bcff-40b9-8182-15f72dde4b26-node-exporter-accelerators-collector-config\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.787601 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.787525 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d4219f4-4ffc-4733-b732-eb264c641245-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-sldj9\" (UID: \"2d4219f4-4ffc-4733-b732-eb264c641245\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" Sep 16 00:35:18.787601 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.787555 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8746e8b6-9c83-440d-a0f9-4b9070ef6104-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-xsrqz\" (UID: \"8746e8b6-9c83-440d-a0f9-4b9070ef6104\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" Sep 16 00:35:18.787601 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.787587 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d4219f4-4ffc-4733-b732-eb264c641245-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-sldj9\" (UID: \"2d4219f4-4ffc-4733-b732-eb264c641245\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" Sep 16 00:35:18.787908 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.787614 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca98b25c-bcff-40b9-8182-15f72dde4b26-node-exporter-textfile\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.787908 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.787648 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca98b25c-bcff-40b9-8182-15f72dde4b26-metrics-client-ca\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.787908 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.787679 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8746e8b6-9c83-440d-a0f9-4b9070ef6104-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-xsrqz\" (UID: \"8746e8b6-9c83-440d-a0f9-4b9070ef6104\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" Sep 16 00:35:18.787908 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.787692 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca98b25c-bcff-40b9-8182-15f72dde4b26-sys\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.787908 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.787715 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdvfb\" (UniqueName: \"kubernetes.io/projected/8746e8b6-9c83-440d-a0f9-4b9070ef6104-kube-api-access-zdvfb\") pod \"openshift-state-metrics-67dcf8c979-xsrqz\" (UID: \"8746e8b6-9c83-440d-a0f9-4b9070ef6104\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" Sep 16 00:35:18.787908 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.787746 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2d4219f4-4ffc-4733-b732-eb264c641245-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-sldj9\" (UID: \"2d4219f4-4ffc-4733-b732-eb264c641245\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" Sep 16 00:35:18.788887 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.788299 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca98b25c-bcff-40b9-8182-15f72dde4b26-node-exporter-accelerators-collector-config\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.788887 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.788373 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca98b25c-bcff-40b9-8182-15f72dde4b26-sys\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.788887 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.788390 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca98b25c-bcff-40b9-8182-15f72dde4b26-node-exporter-textfile\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.788887 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.788425 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2d4219f4-4ffc-4733-b732-eb264c641245-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-sldj9\" (UID: \"2d4219f4-4ffc-4733-b732-eb264c641245\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" Sep 16 00:35:18.788887 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:35:18.788713 2567 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 16 00:35:18.788887 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:35:18.788781 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8746e8b6-9c83-440d-a0f9-4b9070ef6104-openshift-state-metrics-tls podName:8746e8b6-9c83-440d-a0f9-4b9070ef6104 nodeName:}" failed. No retries permitted until 2026-09-16 00:35:19.288763088 +0000 UTC m=+69.650986612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8746e8b6-9c83-440d-a0f9-4b9070ef6104-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-xsrqz" (UID: "8746e8b6-9c83-440d-a0f9-4b9070ef6104") : secret "openshift-state-metrics-tls" not found Sep 16 00:35:18.788887 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.788832 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca98b25c-bcff-40b9-8182-15f72dde4b26-metrics-client-ca\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.789487 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.789461 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d4219f4-4ffc-4733-b732-eb264c641245-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-sldj9\" (UID: \"2d4219f4-4ffc-4733-b732-eb264c641245\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" Sep 16 00:35:18.789848 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.789824 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca98b25c-bcff-40b9-8182-15f72dde4b26-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.790651 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.790631 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca98b25c-bcff-40b9-8182-15f72dde4b26-node-exporter-tls\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.791059 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.791009 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4219f4-4ffc-4733-b732-eb264c641245-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-sldj9\" (UID: \"2d4219f4-4ffc-4733-b732-eb264c641245\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" Sep 16 00:35:18.791120 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.791061 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8746e8b6-9c83-440d-a0f9-4b9070ef6104-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-xsrqz\" (UID: \"8746e8b6-9c83-440d-a0f9-4b9070ef6104\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" Sep 16 00:35:18.792381 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.792319 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d4219f4-4ffc-4733-b732-eb264c641245-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-sldj9\" (UID: \"2d4219f4-4ffc-4733-b732-eb264c641245\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" Sep 16 00:35:18.795645 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.795626 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9z69\" (UniqueName: \"kubernetes.io/projected/ca98b25c-bcff-40b9-8182-15f72dde4b26-kube-api-access-j9z69\") pod \"node-exporter-h8w5x\" (UID: \"ca98b25c-bcff-40b9-8182-15f72dde4b26\") " pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.796075 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.796050 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z68z\" (UniqueName: \"kubernetes.io/projected/2d4219f4-4ffc-4733-b732-eb264c641245-kube-api-access-9z68z\") pod \"kube-state-metrics-59fbdd77d-sldj9\" (UID: \"2d4219f4-4ffc-4733-b732-eb264c641245\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" Sep 16 00:35:18.797268 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.797247 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdvfb\" (UniqueName: \"kubernetes.io/projected/8746e8b6-9c83-440d-a0f9-4b9070ef6104-kube-api-access-zdvfb\") pod \"openshift-state-metrics-67dcf8c979-xsrqz\" (UID: \"8746e8b6-9c83-440d-a0f9-4b9070ef6104\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" Sep 16 00:35:18.810734 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.810711 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:18.910470 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.910439 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:18.918113 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.918077 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h8w5x" Sep 16 00:35:18.928961 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.928944 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" Sep 16 00:35:18.964326 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:18.964204 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:19.291753 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:19.291670 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8746e8b6-9c83-440d-a0f9-4b9070ef6104-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-xsrqz\" (UID: \"8746e8b6-9c83-440d-a0f9-4b9070ef6104\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" Sep 16 00:35:19.292258 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:35:19.291863 2567 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 16 00:35:19.292258 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:35:19.291942 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8746e8b6-9c83-440d-a0f9-4b9070ef6104-openshift-state-metrics-tls podName:8746e8b6-9c83-440d-a0f9-4b9070ef6104 nodeName:}" failed. No retries permitted until 2026-09-16 00:35:20.29192005 +0000 UTC m=+70.654143574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8746e8b6-9c83-440d-a0f9-4b9070ef6104-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-xsrqz" (UID: "8746e8b6-9c83-440d-a0f9-4b9070ef6104") : secret "openshift-state-metrics-tls" not found Sep 16 00:35:19.392052 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:19.391984 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:19.491059 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:19.490995 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:19.586885 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:19.586818 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:19.656210 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:19.656180 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:19.897734 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:19.897663 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:20.298185 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:20.298097 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8746e8b6-9c83-440d-a0f9-4b9070ef6104-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-xsrqz\" (UID: \"8746e8b6-9c83-440d-a0f9-4b9070ef6104\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" Sep 16 00:35:20.301114 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:20.301084 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8746e8b6-9c83-440d-a0f9-4b9070ef6104-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-xsrqz\" (UID: \"8746e8b6-9c83-440d-a0f9-4b9070ef6104\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" Sep 16 00:35:20.337001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:20.336971 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79b88569f4-xrrcb" Sep 16 00:35:20.395341 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:20.395104 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" Sep 16 00:35:20.541936 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:20.541903 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:20.641927 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:20.641890 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:21.049591 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:35:21.049566 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca98b25c_bcff_40b9_8182_15f72dde4b26.slice/crio-feca4860d9750becc6a47cc64f2d4463a8c9ed85c889971316e540e910bf38df WatchSource:0}: Error finding container feca4860d9750becc6a47cc64f2d4463a8c9ed85c889971316e540e910bf38df: Status 404 returned error can't find the container with id feca4860d9750becc6a47cc64f2d4463a8c9ed85c889971316e540e910bf38df Sep 16 00:35:21.080615 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:21.080522 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:21.139364 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:21.139330 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:21.330672 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:21.330639 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz"] Sep 16 00:35:21.333508 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:21.333485 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9"] Sep 16 00:35:21.334144 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:35:21.334116 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8746e8b6_9c83_440d_a0f9_4b9070ef6104.slice/crio-775d8f4c0fe1e5ef699ff153ebd0eb16c1153f5e047899e91bdd824d269deaae WatchSource:0}: Error finding container 775d8f4c0fe1e5ef699ff153ebd0eb16c1153f5e047899e91bdd824d269deaae: Status 404 returned error can't find the container with id 775d8f4c0fe1e5ef699ff153ebd0eb16c1153f5e047899e91bdd824d269deaae Sep 16 00:35:21.336747 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:35:21.336727 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4219f4_4ffc_4733_b732_eb264c641245.slice/crio-038d6795d8e319a106c871a3ffdf00eb299034cc5dfe806fc121d6eadfbdd67f WatchSource:0}: Error finding container 038d6795d8e319a106c871a3ffdf00eb299034cc5dfe806fc121d6eadfbdd67f: Status 404 returned error can't find the container with id 038d6795d8e319a106c871a3ffdf00eb299034cc5dfe806fc121d6eadfbdd67f Sep 16 00:35:21.418519 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:21.418489 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gw589" event={"ID":"f6c0e1cf-272e-412e-b846-d6ee5fc96e28","Type":"ContainerStarted","Data":"26271723c61582e6aa2ae9c1445183e0787721e57400bac4bfe1d4370cf9a55b"} Sep 16 00:35:21.418618 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:21.418600 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:35:21.421581 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:21.421556 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" event={"ID":"8746e8b6-9c83-440d-a0f9-4b9070ef6104","Type":"ContainerStarted","Data":"ffbae82bb03efbfd4c5869cee38cdabd3f6f0b1545937e55e279c139bd6239d4"} Sep 16 00:35:21.421704 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:21.421608 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" event={"ID":"8746e8b6-9c83-440d-a0f9-4b9070ef6104","Type":"ContainerStarted","Data":"775d8f4c0fe1e5ef699ff153ebd0eb16c1153f5e047899e91bdd824d269deaae"} Sep 16 00:35:21.422765 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:21.422745 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" event={"ID":"2d4219f4-4ffc-4733-b732-eb264c641245","Type":"ContainerStarted","Data":"038d6795d8e319a106c871a3ffdf00eb299034cc5dfe806fc121d6eadfbdd67f"} Sep 16 00:35:21.424318 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:21.424299 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fpm24" event={"ID":"be34ccf1-b7da-47ab-9134-2813a37e07a1","Type":"ContainerStarted","Data":"62410f2c64b16b45a5c21e1a1fb2734f5ad46bed52086902b41addd67afc7243"} Sep 16 00:35:21.425512 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:21.425494 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h8w5x" event={"ID":"ca98b25c-bcff-40b9-8182-15f72dde4b26","Type":"ContainerStarted","Data":"feca4860d9750becc6a47cc64f2d4463a8c9ed85c889971316e540e910bf38df"} Sep 16 00:35:21.432964 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:21.432931 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gw589" podStartSLOduration=66.816894129 podStartE2EDuration="1m11.432918655s" totalImagesPullingTime="4.616024526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:21.432703053 +0000 UTC m=+71.794926603" watchObservedRunningTime="2026-09-16 00:35:21.432918655 +0000 UTC m=+71.795142187" Sep 16 00:35:21.446130 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:21.446087 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fpm24" podStartSLOduration=66.605990349 podStartE2EDuration="1m11.44607207s" totalImagesPullingTime="4.840081721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:21.446023016 +0000 UTC m=+71.808246575" watchObservedRunningTime="2026-09-16 00:35:21.44607207 +0000 UTC m=+71.808295604" Sep 16 00:35:21.528359 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:21.528324 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:21.585942 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:21.585868 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:21.651422 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:21.651392 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:21.741950 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:21.741920 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:22.431464 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:22.431426 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" event={"ID":"8746e8b6-9c83-440d-a0f9-4b9070ef6104","Type":"ContainerStarted","Data":"a46e3c29701ce0d9cf40686666f0068d35b97e378ac807154470a02f155e6817"} Sep 16 00:35:22.433334 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:22.433303 2567 generic.go:356] "Generic (PLEG): container finished" podID="ca98b25c-bcff-40b9-8182-15f72dde4b26" containerID="1ce144ec4fd4fa1aece3d457610f2c185e29c1eda82c52e20ce1b6190416772b" exitCode=0 Sep 16 00:35:22.433466 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:22.433345 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h8w5x" event={"ID":"ca98b25c-bcff-40b9-8182-15f72dde4b26","Type":"ContainerDied","Data":"1ce144ec4fd4fa1aece3d457610f2c185e29c1eda82c52e20ce1b6190416772b"} Sep 16 00:35:23.386885 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.386856 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:23.433984 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.433955 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55d79c8df9-rqb5m"] Sep 16 00:35:23.436552 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.436531 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.439249 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.439222 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 16 00:35:23.439358 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.439257 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 16 00:35:23.439358 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.439222 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 16 00:35:23.439462 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.439361 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 16 00:35:23.439732 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.439704 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" event={"ID":"8746e8b6-9c83-440d-a0f9-4b9070ef6104","Type":"ContainerStarted","Data":"6dc507a370c98fb68e9d6d206585270d4cd7a591d57f436d5ee906a8e064d91a"} Sep 16 00:35:23.440662 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.440495 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 16 00:35:23.440662 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.440575 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 16 00:35:23.441031 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.441002 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 16 00:35:23.441164 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.441081 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fwwz4\"" Sep 16 00:35:23.442676 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.442654 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" event={"ID":"2d4219f4-4ffc-4733-b732-eb264c641245","Type":"ContainerStarted","Data":"7548c43774066f0f75f96c8fa0a31d7903f5522f3f365ecca2e41b8412fd4247"} Sep 16 00:35:23.442776 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.442708 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" event={"ID":"2d4219f4-4ffc-4733-b732-eb264c641245","Type":"ContainerStarted","Data":"5799de50365f365e0cba66699adb1064cce80685b5c2f4bf03ae5744f061ca02"} Sep 16 00:35:23.442776 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.442743 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" event={"ID":"2d4219f4-4ffc-4733-b732-eb264c641245","Type":"ContainerStarted","Data":"90368c47c8c35bfd5ee80dd5df16e7dd5a223eecae77b6d7b16d8301853cbb52"} Sep 16 00:35:23.445534 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.445514 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 16 00:35:23.447458 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.446866 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h8w5x" event={"ID":"ca98b25c-bcff-40b9-8182-15f72dde4b26","Type":"ContainerStarted","Data":"b0473c2234e5d5907eba4f42244ef6e2a5330ca5c012838549aaa2e04d29d2db"} Sep 16 00:35:23.447458 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.446911 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h8w5x" event={"ID":"ca98b25c-bcff-40b9-8182-15f72dde4b26","Type":"ContainerStarted","Data":"e6c50eb62c9455026913f0cddfeb0ffa3c604ddbac733c32889e78a47ea29223"} Sep 16 00:35:23.450822 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.450798 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55d79c8df9-rqb5m"] Sep 16 00:35:23.469512 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.465452 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:23.478084 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.478013 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h8w5x" podStartSLOduration=4.511724115 podStartE2EDuration="5.477997257s" totalImagesPullingTime="888.210533ms" totalInitContainerRuntime="78.062609ms" isStatefulPod=true podCreationTimestamp="2026-09-16 00:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:23.477796 +0000 UTC m=+73.840019577" watchObservedRunningTime="2026-09-16 00:35:23.477997257 +0000 UTC m=+73.840220786" Sep 16 00:35:23.495657 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.495615 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xsrqz" podStartSLOduration=4.043272503 podStartE2EDuration="5.495601194s" totalImagesPullingTime="1.452328691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:23.494697936 +0000 UTC m=+73.856921498" watchObservedRunningTime="2026-09-16 00:35:23.495601194 +0000 UTC m=+73.857824726" Sep 16 00:35:23.512540 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.512507 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sldj9" podStartSLOduration=3.944252736 podStartE2EDuration="5.512494539s" totalImagesPullingTime="1.568241803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:23.51165876 +0000 UTC m=+73.873882311" watchObservedRunningTime="2026-09-16 00:35:23.512494539 +0000 UTC m=+73.874718071" Sep 16 00:35:23.526375 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.526352 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/157db980-f502-4b8d-bca8-80dad474f593-console-serving-cert\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.526594 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.526571 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2kkm\" (UniqueName: \"kubernetes.io/projected/157db980-f502-4b8d-bca8-80dad474f593-kube-api-access-g2kkm\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.526668 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.526646 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-console-config\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.526720 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.526678 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/157db980-f502-4b8d-bca8-80dad474f593-console-oauth-config\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.526780 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.526764 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-oauth-serving-cert\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.526852 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.526834 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-service-ca\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.526966 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.526948 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-trusted-ca-bundle\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.627224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.627201 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-trusted-ca-bundle\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.627330 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.627227 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/157db980-f502-4b8d-bca8-80dad474f593-console-serving-cert\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.627330 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.627252 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2kkm\" (UniqueName: \"kubernetes.io/projected/157db980-f502-4b8d-bca8-80dad474f593-kube-api-access-g2kkm\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.627330 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.627270 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-console-config\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.627474 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.627344 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/157db980-f502-4b8d-bca8-80dad474f593-console-oauth-config\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.627474 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.627393 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-oauth-serving-cert\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.627474 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.627424 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-service-ca\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.627961 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.627941 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-console-config\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.628206 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.628185 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-trusted-ca-bundle\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.628253 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.628183 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-oauth-serving-cert\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.628336 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.628316 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-service-ca\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.629844 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.629815 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/157db980-f502-4b8d-bca8-80dad474f593-console-serving-cert\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.629960 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.629940 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/157db980-f502-4b8d-bca8-80dad474f593-console-oauth-config\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.634879 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.634859 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2kkm\" (UniqueName: \"kubernetes.io/projected/157db980-f502-4b8d-bca8-80dad474f593-kube-api-access-g2kkm\") pod \"console-55d79c8df9-rqb5m\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:23.750068 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:23.749993 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:24.019891 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:35:24.019860 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157db980_f502_4b8d_bca8_80dad474f593.slice/crio-e8f1004d1cff24af1dc50070424b538886abf3dc54213cc3da4cbec8865fbfb7 WatchSource:0}: Error finding container e8f1004d1cff24af1dc50070424b538886abf3dc54213cc3da4cbec8865fbfb7: Status 404 returned error can't find the container with id e8f1004d1cff24af1dc50070424b538886abf3dc54213cc3da4cbec8865fbfb7 Sep 16 00:35:24.021711 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:24.021688 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55d79c8df9-rqb5m"] Sep 16 00:35:24.356112 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:24.356079 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:24.452251 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:24.452215 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55d79c8df9-rqb5m" event={"ID":"157db980-f502-4b8d-bca8-80dad474f593","Type":"ContainerStarted","Data":"e8f1004d1cff24af1dc50070424b538886abf3dc54213cc3da4cbec8865fbfb7"} Sep 16 00:35:24.837411 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:24.837378 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:27.467407 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:27.467374 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55d79c8df9-rqb5m" event={"ID":"157db980-f502-4b8d-bca8-80dad474f593","Type":"ContainerStarted","Data":"424c864440f316bd8f511fcfbd46f2af1b0f492b942d89f4f0a2f8de11b51277"} Sep 16 00:35:27.484793 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:27.484745 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55d79c8df9-rqb5m" podStartSLOduration=1.6589608569999998 podStartE2EDuration="4.484729501s" totalImagesPullingTime="2.825768644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:27.483527658 +0000 UTC m=+77.845751227" watchObservedRunningTime="2026-09-16 00:35:27.484729501 +0000 UTC m=+77.846953032" Sep 16 00:35:29.121503 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:29.121475 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:29.173796 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:29.173769 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:30.070750 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:30.070719 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:30.140466 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:30.140441 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:33.122057 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:33.121929 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:33.190418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:33.190384 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:33.750634 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:33.750607 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:33.750777 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:33.750661 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:33.755167 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:33.755146 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:34.386690 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:34.386654 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:34.451961 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:34.451934 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:34.495638 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:34.495615 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:35:34.665090 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:34.664999 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:34.777729 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:34.777496 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:34.873716 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:34.873681 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:39.096720 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:39.096689 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:39.197415 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:39.197388 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:41.492810 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:41.492773 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:41.585976 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:41.585936 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:41.635402 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:41.635376 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:42.190285 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:42.190250 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:42.290789 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:42.290759 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:49.380580 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:49.380519 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:49.437134 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:49.437099 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:35:52.436220 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:35:52.436195 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gw589" Sep 16 00:36:04.381010 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:04.380955 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:36:04.481232 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:04.481203 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:36:37.836965 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:37.836933 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:36:38.069848 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:38.069782 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:36:42.932978 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:42.932935 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6"] Sep 16 00:36:42.935161 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:42.935140 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:42.938378 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:42.938356 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 16 00:36:42.938564 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:42.938539 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 16 00:36:42.938873 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:42.938849 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 16 00:36:42.938995 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:42.938854 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 16 00:36:42.939150 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:42.939133 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8t9hk\"" Sep 16 00:36:42.939543 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:42.939528 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 16 00:36:42.944476 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:42.944461 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 16 00:36:42.944557 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:42.944506 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6"] Sep 16 00:36:43.080913 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.080876 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/128bc9a9-1408-401b-a216-64f8a7197189-serving-certs-ca-bundle\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.080913 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.080916 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/128bc9a9-1408-401b-a216-64f8a7197189-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.081163 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.080933 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/128bc9a9-1408-401b-a216-64f8a7197189-secret-telemeter-client\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.081163 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.080948 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/128bc9a9-1408-401b-a216-64f8a7197189-federate-client-tls\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.081163 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.081004 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/128bc9a9-1408-401b-a216-64f8a7197189-telemeter-client-tls\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.081163 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.081018 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/128bc9a9-1408-401b-a216-64f8a7197189-metrics-client-ca\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.081163 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.081063 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/128bc9a9-1408-401b-a216-64f8a7197189-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.081163 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.081110 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mpwj\" (UniqueName: \"kubernetes.io/projected/128bc9a9-1408-401b-a216-64f8a7197189-kube-api-access-7mpwj\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.181771 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.181744 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/128bc9a9-1408-401b-a216-64f8a7197189-serving-certs-ca-bundle\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.181771 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.181772 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/128bc9a9-1408-401b-a216-64f8a7197189-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.181942 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.181792 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/128bc9a9-1408-401b-a216-64f8a7197189-secret-telemeter-client\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.181942 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.181813 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/128bc9a9-1408-401b-a216-64f8a7197189-federate-client-tls\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.181942 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.181843 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/128bc9a9-1408-401b-a216-64f8a7197189-telemeter-client-tls\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.181942 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.181867 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/128bc9a9-1408-401b-a216-64f8a7197189-metrics-client-ca\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.181942 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.181893 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/128bc9a9-1408-401b-a216-64f8a7197189-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.181942 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.181939 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mpwj\" (UniqueName: \"kubernetes.io/projected/128bc9a9-1408-401b-a216-64f8a7197189-kube-api-access-7mpwj\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.182587 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.182562 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/128bc9a9-1408-401b-a216-64f8a7197189-metrics-client-ca\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.182698 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.182577 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/128bc9a9-1408-401b-a216-64f8a7197189-serving-certs-ca-bundle\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.183080 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.182999 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/128bc9a9-1408-401b-a216-64f8a7197189-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.184468 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.184442 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/128bc9a9-1408-401b-a216-64f8a7197189-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.184742 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.184727 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/128bc9a9-1408-401b-a216-64f8a7197189-federate-client-tls\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.184824 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.184802 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/128bc9a9-1408-401b-a216-64f8a7197189-telemeter-client-tls\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.184881 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.184863 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/128bc9a9-1408-401b-a216-64f8a7197189-secret-telemeter-client\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.190202 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.190179 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mpwj\" (UniqueName: \"kubernetes.io/projected/128bc9a9-1408-401b-a216-64f8a7197189-kube-api-access-7mpwj\") pod \"telemeter-client-5b7ffbcf74-jr2c6\" (UID: \"128bc9a9-1408-401b-a216-64f8a7197189\") " pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.246989 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.246966 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" Sep 16 00:36:43.523302 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.523199 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6"] Sep 16 00:36:43.526008 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:36:43.525980 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128bc9a9_1408_401b_a216_64f8a7197189.slice/crio-18f291803dc21a0fa2c277c39173fbffb60f72cb70827abc969720fde08c6717 WatchSource:0}: Error finding container 18f291803dc21a0fa2c277c39173fbffb60f72cb70827abc969720fde08c6717: Status 404 returned error can't find the container with id 18f291803dc21a0fa2c277c39173fbffb60f72cb70827abc969720fde08c6717 Sep 16 00:36:43.726282 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:43.726254 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" event={"ID":"128bc9a9-1408-401b-a216-64f8a7197189","Type":"ContainerStarted","Data":"18f291803dc21a0fa2c277c39173fbffb60f72cb70827abc969720fde08c6717"} Sep 16 00:36:45.735333 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:45.735254 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" event={"ID":"128bc9a9-1408-401b-a216-64f8a7197189","Type":"ContainerStarted","Data":"cc2de69dab6700f0641b5fc0ea7f6bcd43ac58907f636d47fc88bf1a3075383f"} Sep 16 00:36:46.743376 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:46.743349 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" event={"ID":"128bc9a9-1408-401b-a216-64f8a7197189","Type":"ContainerStarted","Data":"9918ad7f216be153126dd51d996e3761ef799a7980aa239e7dc6b6bde8cae643"} Sep 16 00:36:46.743683 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:46.743406 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" event={"ID":"128bc9a9-1408-401b-a216-64f8a7197189","Type":"ContainerStarted","Data":"3c1b2bb1027c7ed309ad4e477bd11508207a6a0c7cf3e84899340660f3fb446e"} Sep 16 00:36:46.764903 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:46.764836 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5b7ffbcf74-jr2c6" podStartSLOduration=1.732859603 podStartE2EDuration="4.764816036s" totalImagesPullingTime="3.031956433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:36:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:36:46.763420048 +0000 UTC m=+157.125643600" watchObservedRunningTime="2026-09-16 00:36:46.764816036 +0000 UTC m=+157.127039568" Sep 16 00:36:46.839957 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:46.839931 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-675f487c49-nbk6q"] Sep 16 00:36:46.842149 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:46.842130 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:46.854222 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:46.854199 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-675f487c49-nbk6q"] Sep 16 00:36:46.902782 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:46.902723 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:36:47.005354 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.005321 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:36:47.012384 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.012360 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32816639-b59a-4fbf-8d52-4c06c809a291-console-serving-cert\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.012495 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.012400 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-console-config\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.012495 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.012437 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-oauth-serving-cert\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.012495 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.012470 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-service-ca\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.012596 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.012503 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh8kq\" (UniqueName: \"kubernetes.io/projected/32816639-b59a-4fbf-8d52-4c06c809a291-kube-api-access-mh8kq\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.012596 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.012535 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32816639-b59a-4fbf-8d52-4c06c809a291-console-oauth-config\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.012596 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.012558 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-trusted-ca-bundle\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.104651 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.104611 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:36:47.113408 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.113375 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-service-ca\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.113539 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.113415 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh8kq\" (UniqueName: \"kubernetes.io/projected/32816639-b59a-4fbf-8d52-4c06c809a291-kube-api-access-mh8kq\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.113539 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.113457 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32816639-b59a-4fbf-8d52-4c06c809a291-console-oauth-config\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.113539 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.113482 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-trusted-ca-bundle\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.113539 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.113526 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32816639-b59a-4fbf-8d52-4c06c809a291-console-serving-cert\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.113758 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.113569 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-console-config\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.113758 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.113616 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-oauth-serving-cert\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.114216 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.114193 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-service-ca\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.114440 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.114390 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-oauth-serving-cert\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.114440 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.114429 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-console-config\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.114701 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.114675 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-trusted-ca-bundle\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.116462 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.116428 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32816639-b59a-4fbf-8d52-4c06c809a291-console-oauth-config\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.118706 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.118685 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32816639-b59a-4fbf-8d52-4c06c809a291-console-serving-cert\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.122505 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.122468 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh8kq\" (UniqueName: \"kubernetes.io/projected/32816639-b59a-4fbf-8d52-4c06c809a291-kube-api-access-mh8kq\") pod \"console-675f487c49-nbk6q\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.150324 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.150293 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:36:47.426095 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.426001 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-675f487c49-nbk6q"] Sep 16 00:36:47.428832 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:36:47.428795 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32816639_b59a_4fbf_8d52_4c06c809a291.slice/crio-636f626321827bb55ddbf7e57a63bd009d51808b14577a21d7c5ef89b5dbeec4 WatchSource:0}: Error finding container 636f626321827bb55ddbf7e57a63bd009d51808b14577a21d7c5ef89b5dbeec4: Status 404 returned error can't find the container with id 636f626321827bb55ddbf7e57a63bd009d51808b14577a21d7c5ef89b5dbeec4 Sep 16 00:36:47.639927 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.639877 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:36:47.692125 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.692052 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:36:47.748796 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.748767 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-675f487c49-nbk6q" event={"ID":"32816639-b59a-4fbf-8d52-4c06c809a291","Type":"ContainerStarted","Data":"785c17d169f7a213667ab2cfae58fc21bb5dc8c6eda8a93abdf19e66c243bd4a"} Sep 16 00:36:47.749114 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.748817 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-675f487c49-nbk6q" event={"ID":"32816639-b59a-4fbf-8d52-4c06c809a291","Type":"ContainerStarted","Data":"636f626321827bb55ddbf7e57a63bd009d51808b14577a21d7c5ef89b5dbeec4"} Sep 16 00:36:47.767504 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:47.767451 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-675f487c49-nbk6q" podStartSLOduration=1.767432828 podStartE2EDuration="1.767432828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:36:47.766341831 +0000 UTC m=+158.128565387" watchObservedRunningTime="2026-09-16 00:36:47.767432828 +0000 UTC m=+158.129656361" Sep 16 00:36:53.061262 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.061224 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:36:53.161623 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.161594 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:36:53.641355 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.641324 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-675f487c49-nbk6q"] Sep 16 00:36:53.670943 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.670916 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ff79b7c9f-qhnnv"] Sep 16 00:36:53.673216 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.673197 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.684113 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.684092 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ff79b7c9f-qhnnv"] Sep 16 00:36:53.757079 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.757025 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-oauth-serving-cert\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.757214 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.757110 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94b239b5-f924-40c3-a828-85a7af826188-console-serving-cert\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.757214 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.757152 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv9zv\" (UniqueName: \"kubernetes.io/projected/94b239b5-f924-40c3-a828-85a7af826188-kube-api-access-bv9zv\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.757214 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.757175 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-service-ca\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.757214 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.757201 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-trusted-ca-bundle\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.757389 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.757247 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-console-config\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.757389 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.757277 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94b239b5-f924-40c3-a828-85a7af826188-console-oauth-config\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.858429 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.858401 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94b239b5-f924-40c3-a828-85a7af826188-console-oauth-config\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.858429 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.858429 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-oauth-serving-cert\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.858560 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.858461 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94b239b5-f924-40c3-a828-85a7af826188-console-serving-cert\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.858560 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.858486 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv9zv\" (UniqueName: \"kubernetes.io/projected/94b239b5-f924-40c3-a828-85a7af826188-kube-api-access-bv9zv\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.858682 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.858669 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-service-ca\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.858724 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.858701 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-trusted-ca-bundle\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.858779 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.858736 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-console-config\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.859258 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.859235 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-oauth-serving-cert\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.859430 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.859408 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-service-ca\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.859514 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.859498 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-console-config\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.859770 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.859748 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-trusted-ca-bundle\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.861142 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.861114 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94b239b5-f924-40c3-a828-85a7af826188-console-serving-cert\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.861221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.861123 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94b239b5-f924-40c3-a828-85a7af826188-console-oauth-config\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.866663 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.866640 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv9zv\" (UniqueName: \"kubernetes.io/projected/94b239b5-f924-40c3-a828-85a7af826188-kube-api-access-bv9zv\") pod \"console-ff79b7c9f-qhnnv\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:53.981800 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:53.981749 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:36:54.252287 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:54.252216 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ff79b7c9f-qhnnv"] Sep 16 00:36:54.255206 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:36:54.255166 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b239b5_f924_40c3_a828_85a7af826188.slice/crio-0cb6c6452f2c3f1c688d1e431ffd46cc998132ebb9e8f516c113d0728ef08f5c WatchSource:0}: Error finding container 0cb6c6452f2c3f1c688d1e431ffd46cc998132ebb9e8f516c113d0728ef08f5c: Status 404 returned error can't find the container with id 0cb6c6452f2c3f1c688d1e431ffd46cc998132ebb9e8f516c113d0728ef08f5c Sep 16 00:36:54.775009 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:54.774975 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff79b7c9f-qhnnv" event={"ID":"94b239b5-f924-40c3-a828-85a7af826188","Type":"ContainerStarted","Data":"c8205f2027d86863998297295a944fa53139c1faf8295ef57f6ece04d6349c91"} Sep 16 00:36:54.775222 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:54.775027 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff79b7c9f-qhnnv" event={"ID":"94b239b5-f924-40c3-a828-85a7af826188","Type":"ContainerStarted","Data":"0cb6c6452f2c3f1c688d1e431ffd46cc998132ebb9e8f516c113d0728ef08f5c"} Sep 16 00:36:54.792519 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:54.792473 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ff79b7c9f-qhnnv" podStartSLOduration=1.792458555 podStartE2EDuration="1.792458555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:36:54.790585422 +0000 UTC m=+165.152808972" watchObservedRunningTime="2026-09-16 00:36:54.792458555 +0000 UTC m=+165.154682136" Sep 16 00:36:57.150797 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:36:57.150759 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:37:03.982029 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:03.981938 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:37:03.982029 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:03.982002 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:37:03.986876 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:03.986853 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:37:04.814100 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:04.814075 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:37:04.856970 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:04.856936 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55d79c8df9-rqb5m"] Sep 16 00:37:04.925607 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:04.925575 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:37:05.025136 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:05.025105 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:37:18.660854 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:18.660805 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-675f487c49-nbk6q" podUID="32816639-b59a-4fbf-8d52-4c06c809a291" containerName="console" containerID="cri-o://785c17d169f7a213667ab2cfae58fc21bb5dc8c6eda8a93abdf19e66c243bd4a" gracePeriod=15 Sep 16 00:37:18.860700 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:18.860673 2567 generic.go:356] "Generic (PLEG): container finished" podID="32816639-b59a-4fbf-8d52-4c06c809a291" containerID="785c17d169f7a213667ab2cfae58fc21bb5dc8c6eda8a93abdf19e66c243bd4a" exitCode=0 Sep 16 00:37:18.860836 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:18.860714 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-675f487c49-nbk6q" event={"ID":"32816639-b59a-4fbf-8d52-4c06c809a291","Type":"ContainerDied","Data":"785c17d169f7a213667ab2cfae58fc21bb5dc8c6eda8a93abdf19e66c243bd4a"} Sep 16 00:37:18.906483 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:18.902624 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:37:18.934230 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:18.934169 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32816639-b59a-4fbf-8d52-4c06c809a291-console-serving-cert\") pod \"32816639-b59a-4fbf-8d52-4c06c809a291\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " Sep 16 00:37:18.934230 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:18.934211 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-oauth-serving-cert\") pod \"32816639-b59a-4fbf-8d52-4c06c809a291\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " Sep 16 00:37:18.934404 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:18.934259 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-trusted-ca-bundle\") pod \"32816639-b59a-4fbf-8d52-4c06c809a291\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " Sep 16 00:37:18.934404 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:18.934289 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-console-config\") pod \"32816639-b59a-4fbf-8d52-4c06c809a291\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " Sep 16 00:37:18.934404 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:18.934314 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-service-ca\") pod \"32816639-b59a-4fbf-8d52-4c06c809a291\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " Sep 16 00:37:18.934404 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:18.934340 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mh8kq\" (UniqueName: \"kubernetes.io/projected/32816639-b59a-4fbf-8d52-4c06c809a291-kube-api-access-mh8kq\") pod \"32816639-b59a-4fbf-8d52-4c06c809a291\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " Sep 16 00:37:18.934603 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:18.934414 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32816639-b59a-4fbf-8d52-4c06c809a291-console-oauth-config\") pod \"32816639-b59a-4fbf-8d52-4c06c809a291\" (UID: \"32816639-b59a-4fbf-8d52-4c06c809a291\") " Sep 16 00:37:18.934729 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:18.934699 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-console-config" (OuterVolumeSpecName: "console-config") pod "32816639-b59a-4fbf-8d52-4c06c809a291" (UID: "32816639-b59a-4fbf-8d52-4c06c809a291"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:37:18.934729 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:18.934721 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-service-ca" (OuterVolumeSpecName: "service-ca") pod "32816639-b59a-4fbf-8d52-4c06c809a291" (UID: "32816639-b59a-4fbf-8d52-4c06c809a291"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:37:18.934843 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:18.934731 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "32816639-b59a-4fbf-8d52-4c06c809a291" (UID: "32816639-b59a-4fbf-8d52-4c06c809a291"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:37:18.934843 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:18.934708 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "32816639-b59a-4fbf-8d52-4c06c809a291" (UID: "32816639-b59a-4fbf-8d52-4c06c809a291"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:37:18.936667 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:18.936633 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32816639-b59a-4fbf-8d52-4c06c809a291-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "32816639-b59a-4fbf-8d52-4c06c809a291" (UID: "32816639-b59a-4fbf-8d52-4c06c809a291"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 00:37:18.936769 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:18.936718 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32816639-b59a-4fbf-8d52-4c06c809a291-kube-api-access-mh8kq" (OuterVolumeSpecName: "kube-api-access-mh8kq") pod "32816639-b59a-4fbf-8d52-4c06c809a291" (UID: "32816639-b59a-4fbf-8d52-4c06c809a291"). InnerVolumeSpecName "kube-api-access-mh8kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:37:18.936769 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:18.936749 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32816639-b59a-4fbf-8d52-4c06c809a291-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "32816639-b59a-4fbf-8d52-4c06c809a291" (UID: "32816639-b59a-4fbf-8d52-4c06c809a291"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 00:37:19.035325 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:19.035299 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-oauth-serving-cert\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:37:19.035325 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:19.035321 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-trusted-ca-bundle\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:37:19.035325 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:19.035330 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-console-config\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:37:19.035489 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:19.035338 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32816639-b59a-4fbf-8d52-4c06c809a291-service-ca\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:37:19.035489 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:19.035346 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh8kq\" (UniqueName: \"kubernetes.io/projected/32816639-b59a-4fbf-8d52-4c06c809a291-kube-api-access-mh8kq\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:37:19.035489 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:19.035355 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32816639-b59a-4fbf-8d52-4c06c809a291-console-oauth-config\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:37:19.035489 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:19.035363 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32816639-b59a-4fbf-8d52-4c06c809a291-console-serving-cert\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:37:19.422909 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:19.422880 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:37:19.524163 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:19.524139 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:37:19.865621 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:19.865591 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-675f487c49-nbk6q" event={"ID":"32816639-b59a-4fbf-8d52-4c06c809a291","Type":"ContainerDied","Data":"636f626321827bb55ddbf7e57a63bd009d51808b14577a21d7c5ef89b5dbeec4"} Sep 16 00:37:19.866067 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:19.865645 2567 scope.go:117] "RemoveContainer" containerID="785c17d169f7a213667ab2cfae58fc21bb5dc8c6eda8a93abdf19e66c243bd4a" Sep 16 00:37:19.866067 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:19.865660 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-675f487c49-nbk6q" Sep 16 00:37:19.885809 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:19.885787 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-675f487c49-nbk6q"] Sep 16 00:37:19.889968 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:19.889947 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-675f487c49-nbk6q"] Sep 16 00:37:20.119366 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:20.119290 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32816639-b59a-4fbf-8d52-4c06c809a291" path="/var/lib/kubelet/pods/32816639-b59a-4fbf-8d52-4c06c809a291/volumes" Sep 16 00:37:27.123819 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:27.123786 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:37:29.876578 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:29.876514 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55d79c8df9-rqb5m" podUID="157db980-f502-4b8d-bca8-80dad474f593" containerName="console" containerID="cri-o://424c864440f316bd8f511fcfbd46f2af1b0f492b942d89f4f0a2f8de11b51277" gracePeriod=15 Sep 16 00:37:30.110581 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.110561 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:37:30.213403 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.213338 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/157db980-f502-4b8d-bca8-80dad474f593-console-serving-cert\") pod \"157db980-f502-4b8d-bca8-80dad474f593\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " Sep 16 00:37:30.213403 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.213379 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/157db980-f502-4b8d-bca8-80dad474f593-console-oauth-config\") pod \"157db980-f502-4b8d-bca8-80dad474f593\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " Sep 16 00:37:30.213403 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.213401 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-console-config\") pod \"157db980-f502-4b8d-bca8-80dad474f593\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " Sep 16 00:37:30.213620 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.213422 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-service-ca\") pod \"157db980-f502-4b8d-bca8-80dad474f593\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " Sep 16 00:37:30.213620 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.213457 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2kkm\" (UniqueName: \"kubernetes.io/projected/157db980-f502-4b8d-bca8-80dad474f593-kube-api-access-g2kkm\") pod \"157db980-f502-4b8d-bca8-80dad474f593\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " Sep 16 00:37:30.213620 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.213494 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-trusted-ca-bundle\") pod \"157db980-f502-4b8d-bca8-80dad474f593\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " Sep 16 00:37:30.213620 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.213543 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-oauth-serving-cert\") pod \"157db980-f502-4b8d-bca8-80dad474f593\" (UID: \"157db980-f502-4b8d-bca8-80dad474f593\") " Sep 16 00:37:30.213896 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.213864 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-service-ca" (OuterVolumeSpecName: "service-ca") pod "157db980-f502-4b8d-bca8-80dad474f593" (UID: "157db980-f502-4b8d-bca8-80dad474f593"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:37:30.213952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.213885 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "157db980-f502-4b8d-bca8-80dad474f593" (UID: "157db980-f502-4b8d-bca8-80dad474f593"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:37:30.213952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.213872 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-console-config" (OuterVolumeSpecName: "console-config") pod "157db980-f502-4b8d-bca8-80dad474f593" (UID: "157db980-f502-4b8d-bca8-80dad474f593"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:37:30.214016 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.213948 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "157db980-f502-4b8d-bca8-80dad474f593" (UID: "157db980-f502-4b8d-bca8-80dad474f593"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:37:30.215552 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.215533 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/157db980-f502-4b8d-bca8-80dad474f593-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "157db980-f502-4b8d-bca8-80dad474f593" (UID: "157db980-f502-4b8d-bca8-80dad474f593"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 00:37:30.216164 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.216143 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/157db980-f502-4b8d-bca8-80dad474f593-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "157db980-f502-4b8d-bca8-80dad474f593" (UID: "157db980-f502-4b8d-bca8-80dad474f593"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 00:37:30.216215 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.216177 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157db980-f502-4b8d-bca8-80dad474f593-kube-api-access-g2kkm" (OuterVolumeSpecName: "kube-api-access-g2kkm") pod "157db980-f502-4b8d-bca8-80dad474f593" (UID: "157db980-f502-4b8d-bca8-80dad474f593"). InnerVolumeSpecName "kube-api-access-g2kkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:37:30.314640 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.314606 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-trusted-ca-bundle\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:37:30.314640 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.314639 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-oauth-serving-cert\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:37:30.314640 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.314649 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/157db980-f502-4b8d-bca8-80dad474f593-console-serving-cert\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:37:30.314803 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.314658 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/157db980-f502-4b8d-bca8-80dad474f593-console-oauth-config\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:37:30.314803 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.314667 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-console-config\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:37:30.314803 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.314675 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/157db980-f502-4b8d-bca8-80dad474f593-service-ca\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:37:30.314803 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.314682 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2kkm\" (UniqueName: \"kubernetes.io/projected/157db980-f502-4b8d-bca8-80dad474f593-kube-api-access-g2kkm\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:37:30.906976 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.906939 2567 generic.go:356] "Generic (PLEG): container finished" podID="157db980-f502-4b8d-bca8-80dad474f593" containerID="424c864440f316bd8f511fcfbd46f2af1b0f492b942d89f4f0a2f8de11b51277" exitCode=0 Sep 16 00:37:30.907382 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.906992 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55d79c8df9-rqb5m" Sep 16 00:37:30.907382 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.907032 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55d79c8df9-rqb5m" event={"ID":"157db980-f502-4b8d-bca8-80dad474f593","Type":"ContainerDied","Data":"424c864440f316bd8f511fcfbd46f2af1b0f492b942d89f4f0a2f8de11b51277"} Sep 16 00:37:30.907382 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.907131 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55d79c8df9-rqb5m" event={"ID":"157db980-f502-4b8d-bca8-80dad474f593","Type":"ContainerDied","Data":"e8f1004d1cff24af1dc50070424b538886abf3dc54213cc3da4cbec8865fbfb7"} Sep 16 00:37:30.907382 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.907157 2567 scope.go:117] "RemoveContainer" containerID="424c864440f316bd8f511fcfbd46f2af1b0f492b942d89f4f0a2f8de11b51277" Sep 16 00:37:30.915776 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.915760 2567 scope.go:117] "RemoveContainer" containerID="424c864440f316bd8f511fcfbd46f2af1b0f492b942d89f4f0a2f8de11b51277" Sep 16 00:37:30.916052 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:37:30.916019 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"424c864440f316bd8f511fcfbd46f2af1b0f492b942d89f4f0a2f8de11b51277\": container with ID starting with 424c864440f316bd8f511fcfbd46f2af1b0f492b942d89f4f0a2f8de11b51277 not found: ID does not exist" containerID="424c864440f316bd8f511fcfbd46f2af1b0f492b942d89f4f0a2f8de11b51277" Sep 16 00:37:30.916127 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.916069 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"424c864440f316bd8f511fcfbd46f2af1b0f492b942d89f4f0a2f8de11b51277"} err="failed to get container status \"424c864440f316bd8f511fcfbd46f2af1b0f492b942d89f4f0a2f8de11b51277\": rpc error: code = NotFound desc = could not find container \"424c864440f316bd8f511fcfbd46f2af1b0f492b942d89f4f0a2f8de11b51277\": container with ID starting with 424c864440f316bd8f511fcfbd46f2af1b0f492b942d89f4f0a2f8de11b51277 not found: ID does not exist" Sep 16 00:37:30.926997 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.926978 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55d79c8df9-rqb5m"] Sep 16 00:37:30.928755 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:30.928736 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55d79c8df9-rqb5m"] Sep 16 00:37:32.117310 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:32.117273 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="157db980-f502-4b8d-bca8-80dad474f593" path="/var/lib/kubelet/pods/157db980-f502-4b8d-bca8-80dad474f593/volumes" Sep 16 00:37:38.097185 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.097148 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:37:38.197413 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.197367 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:37:38.709212 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.709180 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx"] Sep 16 00:37:38.709485 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.709472 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="157db980-f502-4b8d-bca8-80dad474f593" containerName="console" Sep 16 00:37:38.709526 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.709487 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="157db980-f502-4b8d-bca8-80dad474f593" containerName="console" Sep 16 00:37:38.709526 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.709502 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32816639-b59a-4fbf-8d52-4c06c809a291" containerName="console" Sep 16 00:37:38.709526 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.709507 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="32816639-b59a-4fbf-8d52-4c06c809a291" containerName="console" Sep 16 00:37:38.709612 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.709562 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="157db980-f502-4b8d-bca8-80dad474f593" containerName="console" Sep 16 00:37:38.709612 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.709573 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="32816639-b59a-4fbf-8d52-4c06c809a291" containerName="console" Sep 16 00:37:38.713776 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.713761 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx" Sep 16 00:37:38.716297 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.716278 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gw647\"" Sep 16 00:37:38.716393 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.716284 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 00:37:38.716393 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.716283 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 00:37:38.721109 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.721090 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx"] Sep 16 00:37:38.774997 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.774975 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs475\" (UniqueName: \"kubernetes.io/projected/daa8dcf6-2e52-410f-8cdf-b94260f15f82-kube-api-access-rs475\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx\" (UID: \"daa8dcf6-2e52-410f-8cdf-b94260f15f82\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx" Sep 16 00:37:38.775138 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.775010 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/daa8dcf6-2e52-410f-8cdf-b94260f15f82-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx\" (UID: \"daa8dcf6-2e52-410f-8cdf-b94260f15f82\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx" Sep 16 00:37:38.775138 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.775029 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/daa8dcf6-2e52-410f-8cdf-b94260f15f82-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx\" (UID: \"daa8dcf6-2e52-410f-8cdf-b94260f15f82\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx" Sep 16 00:37:38.876301 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.876261 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs475\" (UniqueName: \"kubernetes.io/projected/daa8dcf6-2e52-410f-8cdf-b94260f15f82-kube-api-access-rs475\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx\" (UID: \"daa8dcf6-2e52-410f-8cdf-b94260f15f82\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx" Sep 16 00:37:38.876425 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.876317 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/daa8dcf6-2e52-410f-8cdf-b94260f15f82-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx\" (UID: \"daa8dcf6-2e52-410f-8cdf-b94260f15f82\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx" Sep 16 00:37:38.876425 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.876346 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/daa8dcf6-2e52-410f-8cdf-b94260f15f82-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx\" (UID: \"daa8dcf6-2e52-410f-8cdf-b94260f15f82\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx" Sep 16 00:37:38.876651 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.876635 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/daa8dcf6-2e52-410f-8cdf-b94260f15f82-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx\" (UID: \"daa8dcf6-2e52-410f-8cdf-b94260f15f82\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx" Sep 16 00:37:38.876692 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.876665 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/daa8dcf6-2e52-410f-8cdf-b94260f15f82-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx\" (UID: \"daa8dcf6-2e52-410f-8cdf-b94260f15f82\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx" Sep 16 00:37:38.884301 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:38.884284 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs475\" (UniqueName: \"kubernetes.io/projected/daa8dcf6-2e52-410f-8cdf-b94260f15f82-kube-api-access-rs475\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx\" (UID: \"daa8dcf6-2e52-410f-8cdf-b94260f15f82\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx" Sep 16 00:37:39.023660 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:39.023588 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx" Sep 16 00:37:39.291272 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:39.291201 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx"] Sep 16 00:37:39.294105 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:37:39.294078 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa8dcf6_2e52_410f_8cdf_b94260f15f82.slice/crio-9113820d3b4169c6304e7d2b235aa412cf3426b2165852253fd0ef556d6e1091 WatchSource:0}: Error finding container 9113820d3b4169c6304e7d2b235aa412cf3426b2165852253fd0ef556d6e1091: Status 404 returned error can't find the container with id 9113820d3b4169c6304e7d2b235aa412cf3426b2165852253fd0ef556d6e1091 Sep 16 00:37:39.940297 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:39.940262 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx" event={"ID":"daa8dcf6-2e52-410f-8cdf-b94260f15f82","Type":"ContainerStarted","Data":"9113820d3b4169c6304e7d2b235aa412cf3426b2165852253fd0ef556d6e1091"} Sep 16 00:37:46.967870 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:46.967836 2567 generic.go:356] "Generic (PLEG): container finished" podID="daa8dcf6-2e52-410f-8cdf-b94260f15f82" containerID="3115d5103a44e4c432da4adf47c6dd386adee1ff99fe5e4ef7f680362c3f09c1" exitCode=0 Sep 16 00:37:46.968310 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:46.967918 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx" event={"ID":"daa8dcf6-2e52-410f-8cdf-b94260f15f82","Type":"ContainerDied","Data":"3115d5103a44e4c432da4adf47c6dd386adee1ff99fe5e4ef7f680362c3f09c1"} Sep 16 00:37:49.980812 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:49.980776 2567 generic.go:356] "Generic (PLEG): container finished" podID="daa8dcf6-2e52-410f-8cdf-b94260f15f82" containerID="a721a05914f745373d8c74a2717aa9462242c932cbc0f7f060a98dd9d5759440" exitCode=0 Sep 16 00:37:49.981284 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:49.980858 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx" event={"ID":"daa8dcf6-2e52-410f-8cdf-b94260f15f82","Type":"ContainerDied","Data":"a721a05914f745373d8c74a2717aa9462242c932cbc0f7f060a98dd9d5759440"} Sep 16 00:37:57.011266 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:57.011232 2567 generic.go:356] "Generic (PLEG): container finished" podID="daa8dcf6-2e52-410f-8cdf-b94260f15f82" containerID="b5262dbbea05497704a7289663d156c1dee8a688f4f29160c90a681a660fccfc" exitCode=0 Sep 16 00:37:57.011641 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:57.011313 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx" event={"ID":"daa8dcf6-2e52-410f-8cdf-b94260f15f82","Type":"ContainerDied","Data":"b5262dbbea05497704a7289663d156c1dee8a688f4f29160c90a681a660fccfc"} Sep 16 00:37:58.134446 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:58.134423 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx" Sep 16 00:37:58.232656 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:58.232626 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/daa8dcf6-2e52-410f-8cdf-b94260f15f82-bundle\") pod \"daa8dcf6-2e52-410f-8cdf-b94260f15f82\" (UID: \"daa8dcf6-2e52-410f-8cdf-b94260f15f82\") " Sep 16 00:37:58.232788 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:58.232672 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/daa8dcf6-2e52-410f-8cdf-b94260f15f82-util\") pod \"daa8dcf6-2e52-410f-8cdf-b94260f15f82\" (UID: \"daa8dcf6-2e52-410f-8cdf-b94260f15f82\") " Sep 16 00:37:58.232844 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:58.232802 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs475\" (UniqueName: \"kubernetes.io/projected/daa8dcf6-2e52-410f-8cdf-b94260f15f82-kube-api-access-rs475\") pod \"daa8dcf6-2e52-410f-8cdf-b94260f15f82\" (UID: \"daa8dcf6-2e52-410f-8cdf-b94260f15f82\") " Sep 16 00:37:58.239168 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:58.233834 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa8dcf6-2e52-410f-8cdf-b94260f15f82-bundle" (OuterVolumeSpecName: "bundle") pod "daa8dcf6-2e52-410f-8cdf-b94260f15f82" (UID: "daa8dcf6-2e52-410f-8cdf-b94260f15f82"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:37:58.239168 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:58.238986 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa8dcf6-2e52-410f-8cdf-b94260f15f82-util" (OuterVolumeSpecName: "util") pod "daa8dcf6-2e52-410f-8cdf-b94260f15f82" (UID: "daa8dcf6-2e52-410f-8cdf-b94260f15f82"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:37:58.239168 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:58.239012 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa8dcf6-2e52-410f-8cdf-b94260f15f82-kube-api-access-rs475" (OuterVolumeSpecName: "kube-api-access-rs475") pod "daa8dcf6-2e52-410f-8cdf-b94260f15f82" (UID: "daa8dcf6-2e52-410f-8cdf-b94260f15f82"). InnerVolumeSpecName "kube-api-access-rs475". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:37:58.334649 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:58.334623 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/daa8dcf6-2e52-410f-8cdf-b94260f15f82-bundle\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:37:58.334649 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:58.334645 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/daa8dcf6-2e52-410f-8cdf-b94260f15f82-util\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:37:58.334780 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:58.334658 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs475\" (UniqueName: \"kubernetes.io/projected/daa8dcf6-2e52-410f-8cdf-b94260f15f82-kube-api-access-rs475\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:37:59.019546 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:59.019521 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx" Sep 16 00:37:59.019546 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:59.019522 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcgqx" event={"ID":"daa8dcf6-2e52-410f-8cdf-b94260f15f82","Type":"ContainerDied","Data":"9113820d3b4169c6304e7d2b235aa412cf3426b2165852253fd0ef556d6e1091"} Sep 16 00:37:59.019785 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:37:59.019594 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9113820d3b4169c6304e7d2b235aa412cf3426b2165852253fd0ef556d6e1091" Sep 16 00:38:04.743332 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:04.743300 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:05.501930 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:05.501890 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:05.621080 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:05.621023 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:06.661926 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:06.661893 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n8xcr"] Sep 16 00:38:06.662298 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:06.662162 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa8dcf6-2e52-410f-8cdf-b94260f15f82" containerName="extract" Sep 16 00:38:06.662298 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:06.662173 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa8dcf6-2e52-410f-8cdf-b94260f15f82" containerName="extract" Sep 16 00:38:06.662298 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:06.662184 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa8dcf6-2e52-410f-8cdf-b94260f15f82" containerName="util" Sep 16 00:38:06.662298 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:06.662190 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa8dcf6-2e52-410f-8cdf-b94260f15f82" containerName="util" Sep 16 00:38:06.662298 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:06.662196 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa8dcf6-2e52-410f-8cdf-b94260f15f82" containerName="pull" Sep 16 00:38:06.662298 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:06.662201 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa8dcf6-2e52-410f-8cdf-b94260f15f82" containerName="pull" Sep 16 00:38:06.662298 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:06.662241 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="daa8dcf6-2e52-410f-8cdf-b94260f15f82" containerName="extract" Sep 16 00:38:06.664013 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:06.663998 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n8xcr" Sep 16 00:38:06.667084 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:06.667062 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 16 00:38:06.667084 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:06.667065 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-m67qb\"" Sep 16 00:38:06.667310 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:06.667296 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 16 00:38:06.676086 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:06.676066 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n8xcr"] Sep 16 00:38:06.792522 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:06.792498 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7082195b-3450-4463-8322-454e9ccd66dc-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-n8xcr\" (UID: \"7082195b-3450-4463-8322-454e9ccd66dc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n8xcr" Sep 16 00:38:06.792643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:06.792541 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwmk6\" (UniqueName: \"kubernetes.io/projected/7082195b-3450-4463-8322-454e9ccd66dc-kube-api-access-xwmk6\") pod \"cert-manager-operator-controller-manager-55c444bf86-n8xcr\" (UID: \"7082195b-3450-4463-8322-454e9ccd66dc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n8xcr" Sep 16 00:38:06.893643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:06.893604 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7082195b-3450-4463-8322-454e9ccd66dc-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-n8xcr\" (UID: \"7082195b-3450-4463-8322-454e9ccd66dc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n8xcr" Sep 16 00:38:06.893738 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:06.893660 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwmk6\" (UniqueName: \"kubernetes.io/projected/7082195b-3450-4463-8322-454e9ccd66dc-kube-api-access-xwmk6\") pod \"cert-manager-operator-controller-manager-55c444bf86-n8xcr\" (UID: \"7082195b-3450-4463-8322-454e9ccd66dc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n8xcr" Sep 16 00:38:06.893976 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:06.893957 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7082195b-3450-4463-8322-454e9ccd66dc-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-n8xcr\" (UID: \"7082195b-3450-4463-8322-454e9ccd66dc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n8xcr" Sep 16 00:38:06.907356 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:06.907335 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwmk6\" (UniqueName: \"kubernetes.io/projected/7082195b-3450-4463-8322-454e9ccd66dc-kube-api-access-xwmk6\") pod \"cert-manager-operator-controller-manager-55c444bf86-n8xcr\" (UID: \"7082195b-3450-4463-8322-454e9ccd66dc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n8xcr" Sep 16 00:38:06.973449 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:06.973401 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n8xcr" Sep 16 00:38:07.270768 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:07.270736 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n8xcr"] Sep 16 00:38:07.273710 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:38:07.273679 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7082195b_3450_4463_8322_454e9ccd66dc.slice/crio-8e81532b75bd0bb513fbc8213b4db32a3f2acb66d285f25400b90c83df057284 WatchSource:0}: Error finding container 8e81532b75bd0bb513fbc8213b4db32a3f2acb66d285f25400b90c83df057284: Status 404 returned error can't find the container with id 8e81532b75bd0bb513fbc8213b4db32a3f2acb66d285f25400b90c83df057284 Sep 16 00:38:08.056735 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:08.056685 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n8xcr" event={"ID":"7082195b-3450-4463-8322-454e9ccd66dc","Type":"ContainerStarted","Data":"8e81532b75bd0bb513fbc8213b4db32a3f2acb66d285f25400b90c83df057284"} Sep 16 00:38:08.348982 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:08.348906 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:08.402777 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:08.402746 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:11.070836 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.070801 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n8xcr" event={"ID":"7082195b-3450-4463-8322-454e9ccd66dc","Type":"ContainerStarted","Data":"2dcac19efb5e517de7892a233b8585dd3e913af0a43057f57629aacc00a76314"} Sep 16 00:38:11.099208 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.099152 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n8xcr" podStartSLOduration=1.847173686 podStartE2EDuration="5.099129834s" totalImagesPullingTime="3.251956148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:38:11.09679129 +0000 UTC m=+241.459014867" watchObservedRunningTime="2026-09-16 00:38:11.099129834 +0000 UTC m=+241.461353367" Sep 16 00:38:11.212965 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.212930 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:11.302617 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.302588 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:11.504230 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.504131 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:11.611456 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.611424 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:11.655135 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.655107 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh"] Sep 16 00:38:11.657248 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.657234 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh" Sep 16 00:38:11.660107 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.660083 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 00:38:11.660227 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.660109 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 00:38:11.660227 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.660133 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gw647\"" Sep 16 00:38:11.671592 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.671567 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh"] Sep 16 00:38:11.830816 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.830791 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e33c3dc9-09de-4f39-b002-1e5f72f736c9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh\" (UID: \"e33c3dc9-09de-4f39-b002-1e5f72f736c9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh" Sep 16 00:38:11.830929 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.830830 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmrc2\" (UniqueName: \"kubernetes.io/projected/e33c3dc9-09de-4f39-b002-1e5f72f736c9-kube-api-access-tmrc2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh\" (UID: \"e33c3dc9-09de-4f39-b002-1e5f72f736c9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh" Sep 16 00:38:11.830929 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.830858 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e33c3dc9-09de-4f39-b002-1e5f72f736c9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh\" (UID: \"e33c3dc9-09de-4f39-b002-1e5f72f736c9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh" Sep 16 00:38:11.931256 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.931226 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e33c3dc9-09de-4f39-b002-1e5f72f736c9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh\" (UID: \"e33c3dc9-09de-4f39-b002-1e5f72f736c9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh" Sep 16 00:38:11.931364 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.931272 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmrc2\" (UniqueName: \"kubernetes.io/projected/e33c3dc9-09de-4f39-b002-1e5f72f736c9-kube-api-access-tmrc2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh\" (UID: \"e33c3dc9-09de-4f39-b002-1e5f72f736c9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh" Sep 16 00:38:11.931364 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.931301 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e33c3dc9-09de-4f39-b002-1e5f72f736c9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh\" (UID: \"e33c3dc9-09de-4f39-b002-1e5f72f736c9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh" Sep 16 00:38:11.931584 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.931567 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e33c3dc9-09de-4f39-b002-1e5f72f736c9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh\" (UID: \"e33c3dc9-09de-4f39-b002-1e5f72f736c9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh" Sep 16 00:38:11.931624 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.931599 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e33c3dc9-09de-4f39-b002-1e5f72f736c9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh\" (UID: \"e33c3dc9-09de-4f39-b002-1e5f72f736c9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh" Sep 16 00:38:11.939346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.939326 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmrc2\" (UniqueName: \"kubernetes.io/projected/e33c3dc9-09de-4f39-b002-1e5f72f736c9-kube-api-access-tmrc2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh\" (UID: \"e33c3dc9-09de-4f39-b002-1e5f72f736c9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh" Sep 16 00:38:11.966320 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:11.966298 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh" Sep 16 00:38:12.147467 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:12.147398 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vmxrw"] Sep 16 00:38:12.149702 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:12.149681 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-vmxrw" Sep 16 00:38:12.152237 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:12.152215 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 16 00:38:12.152348 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:12.152248 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hgvth\"" Sep 16 00:38:12.152348 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:12.152251 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 16 00:38:12.160545 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:12.160524 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vmxrw"] Sep 16 00:38:12.238000 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:12.237976 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh"] Sep 16 00:38:12.240423 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:38:12.240395 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33c3dc9_09de_4f39_b002_1e5f72f736c9.slice/crio-23ed4a092df94cb13fab5343ee4017526febd9fcfad81c5872bfd7f26fc8174b WatchSource:0}: Error finding container 23ed4a092df94cb13fab5343ee4017526febd9fcfad81c5872bfd7f26fc8174b: Status 404 returned error can't find the container with id 23ed4a092df94cb13fab5343ee4017526febd9fcfad81c5872bfd7f26fc8174b Sep 16 00:38:12.335801 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:12.335778 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93f2e85e-a096-4c19-bc51-00f443273bae-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vmxrw\" (UID: \"93f2e85e-a096-4c19-bc51-00f443273bae\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vmxrw" Sep 16 00:38:12.335894 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:12.335825 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k99bv\" (UniqueName: \"kubernetes.io/projected/93f2e85e-a096-4c19-bc51-00f443273bae-kube-api-access-k99bv\") pod \"cert-manager-webhook-6486ffd997-vmxrw\" (UID: \"93f2e85e-a096-4c19-bc51-00f443273bae\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vmxrw" Sep 16 00:38:12.436775 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:12.436723 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k99bv\" (UniqueName: \"kubernetes.io/projected/93f2e85e-a096-4c19-bc51-00f443273bae-kube-api-access-k99bv\") pod \"cert-manager-webhook-6486ffd997-vmxrw\" (UID: \"93f2e85e-a096-4c19-bc51-00f443273bae\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vmxrw" Sep 16 00:38:12.436873 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:12.436777 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93f2e85e-a096-4c19-bc51-00f443273bae-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vmxrw\" (UID: \"93f2e85e-a096-4c19-bc51-00f443273bae\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vmxrw" Sep 16 00:38:12.444991 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:12.444961 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93f2e85e-a096-4c19-bc51-00f443273bae-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vmxrw\" (UID: \"93f2e85e-a096-4c19-bc51-00f443273bae\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vmxrw" Sep 16 00:38:12.445123 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:12.445106 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k99bv\" (UniqueName: \"kubernetes.io/projected/93f2e85e-a096-4c19-bc51-00f443273bae-kube-api-access-k99bv\") pod \"cert-manager-webhook-6486ffd997-vmxrw\" (UID: \"93f2e85e-a096-4c19-bc51-00f443273bae\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vmxrw" Sep 16 00:38:12.466933 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:12.466916 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-vmxrw" Sep 16 00:38:12.736163 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:12.736095 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vmxrw"] Sep 16 00:38:12.738762 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:38:12.738738 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93f2e85e_a096_4c19_bc51_00f443273bae.slice/crio-4320a8f54a2391a1f566e27f47c76ceebc1221ef944ae75263cc38839e2d4925 WatchSource:0}: Error finding container 4320a8f54a2391a1f566e27f47c76ceebc1221ef944ae75263cc38839e2d4925: Status 404 returned error can't find the container with id 4320a8f54a2391a1f566e27f47c76ceebc1221ef944ae75263cc38839e2d4925 Sep 16 00:38:13.080262 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:13.080234 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-vmxrw" event={"ID":"93f2e85e-a096-4c19-bc51-00f443273bae","Type":"ContainerStarted","Data":"4320a8f54a2391a1f566e27f47c76ceebc1221ef944ae75263cc38839e2d4925"} Sep 16 00:38:13.081790 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:13.081768 2567 generic.go:356] "Generic (PLEG): container finished" podID="e33c3dc9-09de-4f39-b002-1e5f72f736c9" containerID="af1ab9a4f0e3b2846e04cf761b689651d143105b51de036c39f34a71e3fefba7" exitCode=0 Sep 16 00:38:13.081889 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:13.081835 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh" event={"ID":"e33c3dc9-09de-4f39-b002-1e5f72f736c9","Type":"ContainerDied","Data":"af1ab9a4f0e3b2846e04cf761b689651d143105b51de036c39f34a71e3fefba7"} Sep 16 00:38:13.081889 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:13.081866 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh" event={"ID":"e33c3dc9-09de-4f39-b002-1e5f72f736c9","Type":"ContainerStarted","Data":"23ed4a092df94cb13fab5343ee4017526febd9fcfad81c5872bfd7f26fc8174b"} Sep 16 00:38:13.399063 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:13.398963 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:13.776679 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:13.776611 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:14.344985 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:14.344945 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:14.399138 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:14.399105 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:17.101259 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:17.101224 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-vmxrw" event={"ID":"93f2e85e-a096-4c19-bc51-00f443273bae","Type":"ContainerStarted","Data":"c2e5cad1e057b6d29b878255e08415cd0bcf194ebee00d99dc264af6d3846ac0"} Sep 16 00:38:17.101660 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:17.101313 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-vmxrw" Sep 16 00:38:17.103058 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:17.103020 2567 generic.go:356] "Generic (PLEG): container finished" podID="e33c3dc9-09de-4f39-b002-1e5f72f736c9" containerID="eb6eb287e7de11f035e3e2d8850885f7cbe674303446a133f33b8d20000940c9" exitCode=0 Sep 16 00:38:17.103177 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:17.103105 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh" event={"ID":"e33c3dc9-09de-4f39-b002-1e5f72f736c9","Type":"ContainerDied","Data":"eb6eb287e7de11f035e3e2d8850885f7cbe674303446a133f33b8d20000940c9"} Sep 16 00:38:17.118564 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:17.118515 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-vmxrw" podStartSLOduration=1.819908984 podStartE2EDuration="5.118500448s" totalImagesPullingTime="3.298591464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:38:17.117952287 +0000 UTC m=+247.480175820" watchObservedRunningTime="2026-09-16 00:38:17.118500448 +0000 UTC m=+247.480723980" Sep 16 00:38:18.108896 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:18.108867 2567 generic.go:356] "Generic (PLEG): container finished" podID="e33c3dc9-09de-4f39-b002-1e5f72f736c9" containerID="b49d139022d35206b3906c234ddfd5975abecff4fb95e631b86016cc128227b0" exitCode=0 Sep 16 00:38:18.109300 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:18.108933 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh" event={"ID":"e33c3dc9-09de-4f39-b002-1e5f72f736c9","Type":"ContainerDied","Data":"b49d139022d35206b3906c234ddfd5975abecff4fb95e631b86016cc128227b0"} Sep 16 00:38:19.230534 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:19.230512 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh" Sep 16 00:38:19.391739 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:19.391665 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmrc2\" (UniqueName: \"kubernetes.io/projected/e33c3dc9-09de-4f39-b002-1e5f72f736c9-kube-api-access-tmrc2\") pod \"e33c3dc9-09de-4f39-b002-1e5f72f736c9\" (UID: \"e33c3dc9-09de-4f39-b002-1e5f72f736c9\") " Sep 16 00:38:19.391739 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:19.391726 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e33c3dc9-09de-4f39-b002-1e5f72f736c9-util\") pod \"e33c3dc9-09de-4f39-b002-1e5f72f736c9\" (UID: \"e33c3dc9-09de-4f39-b002-1e5f72f736c9\") " Sep 16 00:38:19.391892 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:19.391789 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e33c3dc9-09de-4f39-b002-1e5f72f736c9-bundle\") pod \"e33c3dc9-09de-4f39-b002-1e5f72f736c9\" (UID: \"e33c3dc9-09de-4f39-b002-1e5f72f736c9\") " Sep 16 00:38:19.392175 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:19.392145 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33c3dc9-09de-4f39-b002-1e5f72f736c9-bundle" (OuterVolumeSpecName: "bundle") pod "e33c3dc9-09de-4f39-b002-1e5f72f736c9" (UID: "e33c3dc9-09de-4f39-b002-1e5f72f736c9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:38:19.393853 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:19.393828 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33c3dc9-09de-4f39-b002-1e5f72f736c9-kube-api-access-tmrc2" (OuterVolumeSpecName: "kube-api-access-tmrc2") pod "e33c3dc9-09de-4f39-b002-1e5f72f736c9" (UID: "e33c3dc9-09de-4f39-b002-1e5f72f736c9"). InnerVolumeSpecName "kube-api-access-tmrc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:38:19.396183 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:19.396163 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33c3dc9-09de-4f39-b002-1e5f72f736c9-util" (OuterVolumeSpecName: "util") pod "e33c3dc9-09de-4f39-b002-1e5f72f736c9" (UID: "e33c3dc9-09de-4f39-b002-1e5f72f736c9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:38:19.493031 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:19.493008 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e33c3dc9-09de-4f39-b002-1e5f72f736c9-bundle\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:38:19.493149 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:19.493032 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmrc2\" (UniqueName: \"kubernetes.io/projected/e33c3dc9-09de-4f39-b002-1e5f72f736c9-kube-api-access-tmrc2\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:38:19.493149 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:19.493072 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e33c3dc9-09de-4f39-b002-1e5f72f736c9-util\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:38:20.119313 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:20.119286 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh" Sep 16 00:38:20.119512 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:20.119319 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8ngzh" event={"ID":"e33c3dc9-09de-4f39-b002-1e5f72f736c9","Type":"ContainerDied","Data":"23ed4a092df94cb13fab5343ee4017526febd9fcfad81c5872bfd7f26fc8174b"} Sep 16 00:38:20.119512 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:20.119360 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23ed4a092df94cb13fab5343ee4017526febd9fcfad81c5872bfd7f26fc8174b" Sep 16 00:38:21.766686 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:21.766651 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:23.111298 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:23.111266 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-vmxrw" Sep 16 00:38:23.656133 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:23.656103 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:23.756299 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:23.756268 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:26.287394 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:26.287360 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:26.388619 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:26.388585 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:26.488310 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:26.488282 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:28.692308 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:28.692273 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgw99"] Sep 16 00:38:28.692709 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:28.692528 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e33c3dc9-09de-4f39-b002-1e5f72f736c9" containerName="pull" Sep 16 00:38:28.692709 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:28.692539 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e33c3dc9-09de-4f39-b002-1e5f72f736c9" containerName="pull" Sep 16 00:38:28.692709 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:28.692550 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e33c3dc9-09de-4f39-b002-1e5f72f736c9" containerName="extract" Sep 16 00:38:28.692709 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:28.692555 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e33c3dc9-09de-4f39-b002-1e5f72f736c9" containerName="extract" Sep 16 00:38:28.692709 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:28.692572 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e33c3dc9-09de-4f39-b002-1e5f72f736c9" containerName="util" Sep 16 00:38:28.692709 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:28.692578 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e33c3dc9-09de-4f39-b002-1e5f72f736c9" containerName="util" Sep 16 00:38:28.692709 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:28.692622 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e33c3dc9-09de-4f39-b002-1e5f72f736c9" containerName="extract" Sep 16 00:38:28.694370 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:28.694355 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgw99" Sep 16 00:38:28.696982 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:28.696960 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 16 00:38:28.696982 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:28.696974 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 16 00:38:28.697161 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:28.697071 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-qj56d\"" Sep 16 00:38:28.703525 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:28.703504 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgw99"] Sep 16 00:38:28.754015 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:28.753990 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7fz6\" (UniqueName: \"kubernetes.io/projected/5658bcbf-21df-411e-b218-43a8ec27c0d4-kube-api-access-g7fz6\") pod \"openshift-lws-operator-fd8ccff4c-cgw99\" (UID: \"5658bcbf-21df-411e-b218-43a8ec27c0d4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgw99" Sep 16 00:38:28.754154 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:28.754017 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5658bcbf-21df-411e-b218-43a8ec27c0d4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cgw99\" (UID: \"5658bcbf-21df-411e-b218-43a8ec27c0d4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgw99" Sep 16 00:38:28.854775 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:28.854750 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7fz6\" (UniqueName: \"kubernetes.io/projected/5658bcbf-21df-411e-b218-43a8ec27c0d4-kube-api-access-g7fz6\") pod \"openshift-lws-operator-fd8ccff4c-cgw99\" (UID: \"5658bcbf-21df-411e-b218-43a8ec27c0d4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgw99" Sep 16 00:38:28.854885 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:28.854780 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5658bcbf-21df-411e-b218-43a8ec27c0d4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cgw99\" (UID: \"5658bcbf-21df-411e-b218-43a8ec27c0d4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgw99" Sep 16 00:38:28.855171 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:28.855152 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5658bcbf-21df-411e-b218-43a8ec27c0d4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cgw99\" (UID: \"5658bcbf-21df-411e-b218-43a8ec27c0d4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgw99" Sep 16 00:38:28.863802 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:28.863785 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7fz6\" (UniqueName: \"kubernetes.io/projected/5658bcbf-21df-411e-b218-43a8ec27c0d4-kube-api-access-g7fz6\") pod \"openshift-lws-operator-fd8ccff4c-cgw99\" (UID: \"5658bcbf-21df-411e-b218-43a8ec27c0d4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgw99" Sep 16 00:38:29.003723 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:29.003620 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgw99" Sep 16 00:38:29.286497 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:29.286410 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgw99"] Sep 16 00:38:29.289778 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:38:29.289745 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5658bcbf_21df_411e_b218_43a8ec27c0d4.slice/crio-3e55d85d05d41b2c1582aeb5287803265b5d8904eb9a585ccb70dd948537f4af WatchSource:0}: Error finding container 3e55d85d05d41b2c1582aeb5287803265b5d8904eb9a585ccb70dd948537f4af: Status 404 returned error can't find the container with id 3e55d85d05d41b2c1582aeb5287803265b5d8904eb9a585ccb70dd948537f4af Sep 16 00:38:30.160511 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:30.160477 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgw99" event={"ID":"5658bcbf-21df-411e-b218-43a8ec27c0d4","Type":"ContainerStarted","Data":"3e55d85d05d41b2c1582aeb5287803265b5d8904eb9a585ccb70dd948537f4af"} Sep 16 00:38:32.170486 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:32.170447 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgw99" event={"ID":"5658bcbf-21df-411e-b218-43a8ec27c0d4","Type":"ContainerStarted","Data":"3289cb814530328cafd87ffe26b1b6e481590333f165bcd6dd380ef515443729"} Sep 16 00:38:32.189480 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:32.189434 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cgw99" podStartSLOduration=2.172796492 podStartE2EDuration="4.189419995s" totalImagesPullingTime="2.016623503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:38:32.187879088 +0000 UTC m=+262.550102622" watchObservedRunningTime="2026-09-16 00:38:32.189419995 +0000 UTC m=+262.551643526" Sep 16 00:38:33.002210 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:33.002177 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:33.104362 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:33.104333 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:34.782937 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:34.782906 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:35.754835 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:35.754804 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj"] Sep 16 00:38:35.757119 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:35.757098 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj" Sep 16 00:38:35.759919 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:35.759883 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 00:38:35.760054 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:35.759943 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 00:38:35.761010 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:35.760987 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gw647\"" Sep 16 00:38:35.765846 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:35.765825 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj"] Sep 16 00:38:35.811834 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:35.811811 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26a84120-fd42-44c3-99e0-9597316f5644-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj\" (UID: \"26a84120-fd42-44c3-99e0-9597316f5644\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj" Sep 16 00:38:35.812140 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:35.811849 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jbr6\" (UniqueName: \"kubernetes.io/projected/26a84120-fd42-44c3-99e0-9597316f5644-kube-api-access-9jbr6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj\" (UID: \"26a84120-fd42-44c3-99e0-9597316f5644\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj" Sep 16 00:38:35.812140 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:35.811867 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26a84120-fd42-44c3-99e0-9597316f5644-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj\" (UID: \"26a84120-fd42-44c3-99e0-9597316f5644\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj" Sep 16 00:38:35.912832 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:35.912808 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26a84120-fd42-44c3-99e0-9597316f5644-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj\" (UID: \"26a84120-fd42-44c3-99e0-9597316f5644\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj" Sep 16 00:38:35.912931 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:35.912849 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jbr6\" (UniqueName: \"kubernetes.io/projected/26a84120-fd42-44c3-99e0-9597316f5644-kube-api-access-9jbr6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj\" (UID: \"26a84120-fd42-44c3-99e0-9597316f5644\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj" Sep 16 00:38:35.912931 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:35.912868 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26a84120-fd42-44c3-99e0-9597316f5644-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj\" (UID: \"26a84120-fd42-44c3-99e0-9597316f5644\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj" Sep 16 00:38:35.913223 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:35.913208 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26a84120-fd42-44c3-99e0-9597316f5644-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj\" (UID: \"26a84120-fd42-44c3-99e0-9597316f5644\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj" Sep 16 00:38:35.913271 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:35.913248 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26a84120-fd42-44c3-99e0-9597316f5644-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj\" (UID: \"26a84120-fd42-44c3-99e0-9597316f5644\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj" Sep 16 00:38:35.920792 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:35.920770 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jbr6\" (UniqueName: \"kubernetes.io/projected/26a84120-fd42-44c3-99e0-9597316f5644-kube-api-access-9jbr6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj\" (UID: \"26a84120-fd42-44c3-99e0-9597316f5644\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj" Sep 16 00:38:36.067290 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:36.067258 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj" Sep 16 00:38:36.338585 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:36.338509 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj"] Sep 16 00:38:36.343145 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:38:36.343113 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a84120_fd42_44c3_99e0_9597316f5644.slice/crio-3329ac4f2a1d2bda4b256a9561e073ccd92ce1b9c889a8d59ccd33541b98a128 WatchSource:0}: Error finding container 3329ac4f2a1d2bda4b256a9561e073ccd92ce1b9c889a8d59ccd33541b98a128: Status 404 returned error can't find the container with id 3329ac4f2a1d2bda4b256a9561e073ccd92ce1b9c889a8d59ccd33541b98a128 Sep 16 00:38:37.193209 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:37.193126 2567 generic.go:356] "Generic (PLEG): container finished" podID="26a84120-fd42-44c3-99e0-9597316f5644" containerID="d23d42f84814b9069d0a9fca93c0d9b92372f22b4e4eb96ea0c9dfdb23e6f04b" exitCode=0 Sep 16 00:38:37.193209 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:37.193188 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj" event={"ID":"26a84120-fd42-44c3-99e0-9597316f5644","Type":"ContainerDied","Data":"d23d42f84814b9069d0a9fca93c0d9b92372f22b4e4eb96ea0c9dfdb23e6f04b"} Sep 16 00:38:37.193584 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:37.193227 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj" event={"ID":"26a84120-fd42-44c3-99e0-9597316f5644","Type":"ContainerStarted","Data":"3329ac4f2a1d2bda4b256a9561e073ccd92ce1b9c889a8d59ccd33541b98a128"} Sep 16 00:38:38.199156 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:38.199074 2567 generic.go:356] "Generic (PLEG): container finished" podID="26a84120-fd42-44c3-99e0-9597316f5644" containerID="25c2aefe4cb2d22af1fa2d8e4173dffd22799768cd1729b3410108add1578332" exitCode=0 Sep 16 00:38:38.199601 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:38.199155 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj" event={"ID":"26a84120-fd42-44c3-99e0-9597316f5644","Type":"ContainerDied","Data":"25c2aefe4cb2d22af1fa2d8e4173dffd22799768cd1729b3410108add1578332"} Sep 16 00:38:38.561508 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:38.561486 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:39.206211 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:39.206170 2567 generic.go:356] "Generic (PLEG): container finished" podID="26a84120-fd42-44c3-99e0-9597316f5644" containerID="aa95de8f883e9da02753392f5bb641c5959c27c84c2e05d530dfaa1fce0a544f" exitCode=0 Sep 16 00:38:39.206619 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:39.206240 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj" event={"ID":"26a84120-fd42-44c3-99e0-9597316f5644","Type":"ContainerDied","Data":"aa95de8f883e9da02753392f5bb641c5959c27c84c2e05d530dfaa1fce0a544f"} Sep 16 00:38:40.330020 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:40.329996 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj" Sep 16 00:38:40.446334 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:40.446305 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26a84120-fd42-44c3-99e0-9597316f5644-util\") pod \"26a84120-fd42-44c3-99e0-9597316f5644\" (UID: \"26a84120-fd42-44c3-99e0-9597316f5644\") " Sep 16 00:38:40.446484 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:40.446355 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jbr6\" (UniqueName: \"kubernetes.io/projected/26a84120-fd42-44c3-99e0-9597316f5644-kube-api-access-9jbr6\") pod \"26a84120-fd42-44c3-99e0-9597316f5644\" (UID: \"26a84120-fd42-44c3-99e0-9597316f5644\") " Sep 16 00:38:40.446484 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:40.446398 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26a84120-fd42-44c3-99e0-9597316f5644-bundle\") pod \"26a84120-fd42-44c3-99e0-9597316f5644\" (UID: \"26a84120-fd42-44c3-99e0-9597316f5644\") " Sep 16 00:38:40.447152 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:40.447119 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a84120-fd42-44c3-99e0-9597316f5644-bundle" (OuterVolumeSpecName: "bundle") pod "26a84120-fd42-44c3-99e0-9597316f5644" (UID: "26a84120-fd42-44c3-99e0-9597316f5644"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:38:40.448541 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:40.448511 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a84120-fd42-44c3-99e0-9597316f5644-kube-api-access-9jbr6" (OuterVolumeSpecName: "kube-api-access-9jbr6") pod "26a84120-fd42-44c3-99e0-9597316f5644" (UID: "26a84120-fd42-44c3-99e0-9597316f5644"). InnerVolumeSpecName "kube-api-access-9jbr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:38:40.452123 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:40.452007 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a84120-fd42-44c3-99e0-9597316f5644-util" (OuterVolumeSpecName: "util") pod "26a84120-fd42-44c3-99e0-9597316f5644" (UID: "26a84120-fd42-44c3-99e0-9597316f5644"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:38:40.547808 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:40.547746 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26a84120-fd42-44c3-99e0-9597316f5644-util\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:38:40.547808 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:40.547770 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jbr6\" (UniqueName: \"kubernetes.io/projected/26a84120-fd42-44c3-99e0-9597316f5644-kube-api-access-9jbr6\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:38:40.547808 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:40.547782 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26a84120-fd42-44c3-99e0-9597316f5644-bundle\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:38:41.215895 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:41.215871 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj" Sep 16 00:38:41.215895 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:41.215879 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnrpsj" event={"ID":"26a84120-fd42-44c3-99e0-9597316f5644","Type":"ContainerDied","Data":"3329ac4f2a1d2bda4b256a9561e073ccd92ce1b9c889a8d59ccd33541b98a128"} Sep 16 00:38:41.216105 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:41.215919 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3329ac4f2a1d2bda4b256a9561e073ccd92ce1b9c889a8d59ccd33541b98a128" Sep 16 00:38:46.160146 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.160110 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w"] Sep 16 00:38:46.160583 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.160509 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a84120-fd42-44c3-99e0-9597316f5644" containerName="pull" Sep 16 00:38:46.160583 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.160526 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a84120-fd42-44c3-99e0-9597316f5644" containerName="pull" Sep 16 00:38:46.160583 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.160554 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a84120-fd42-44c3-99e0-9597316f5644" containerName="extract" Sep 16 00:38:46.160583 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.160563 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a84120-fd42-44c3-99e0-9597316f5644" containerName="extract" Sep 16 00:38:46.160583 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.160581 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a84120-fd42-44c3-99e0-9597316f5644" containerName="util" Sep 16 00:38:46.160837 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.160588 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a84120-fd42-44c3-99e0-9597316f5644" containerName="util" Sep 16 00:38:46.160837 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.160654 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="26a84120-fd42-44c3-99e0-9597316f5644" containerName="extract" Sep 16 00:38:46.163371 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.163349 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w" Sep 16 00:38:46.165943 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.165913 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 00:38:46.166070 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.165975 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gw647\"" Sep 16 00:38:46.166070 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.166029 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 00:38:46.172723 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.172701 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w"] Sep 16 00:38:46.287829 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.287799 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d7df985-5a6c-451f-8d55-f88d795f0195-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w\" (UID: \"0d7df985-5a6c-451f-8d55-f88d795f0195\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w" Sep 16 00:38:46.287829 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.287831 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7rkl\" (UniqueName: \"kubernetes.io/projected/0d7df985-5a6c-451f-8d55-f88d795f0195-kube-api-access-d7rkl\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w\" (UID: \"0d7df985-5a6c-451f-8d55-f88d795f0195\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w" Sep 16 00:38:46.287985 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.287912 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d7df985-5a6c-451f-8d55-f88d795f0195-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w\" (UID: \"0d7df985-5a6c-451f-8d55-f88d795f0195\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w" Sep 16 00:38:46.388765 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.388744 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d7df985-5a6c-451f-8d55-f88d795f0195-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w\" (UID: \"0d7df985-5a6c-451f-8d55-f88d795f0195\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w" Sep 16 00:38:46.388876 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.388781 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d7df985-5a6c-451f-8d55-f88d795f0195-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w\" (UID: \"0d7df985-5a6c-451f-8d55-f88d795f0195\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w" Sep 16 00:38:46.388876 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.388798 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7rkl\" (UniqueName: \"kubernetes.io/projected/0d7df985-5a6c-451f-8d55-f88d795f0195-kube-api-access-d7rkl\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w\" (UID: \"0d7df985-5a6c-451f-8d55-f88d795f0195\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w" Sep 16 00:38:46.389175 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.389156 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d7df985-5a6c-451f-8d55-f88d795f0195-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w\" (UID: \"0d7df985-5a6c-451f-8d55-f88d795f0195\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w" Sep 16 00:38:46.389246 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.389184 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d7df985-5a6c-451f-8d55-f88d795f0195-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w\" (UID: \"0d7df985-5a6c-451f-8d55-f88d795f0195\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w" Sep 16 00:38:46.397076 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.397027 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7rkl\" (UniqueName: \"kubernetes.io/projected/0d7df985-5a6c-451f-8d55-f88d795f0195-kube-api-access-d7rkl\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w\" (UID: \"0d7df985-5a6c-451f-8d55-f88d795f0195\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w" Sep 16 00:38:46.472764 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.472713 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w" Sep 16 00:38:46.743355 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:46.743333 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w"] Sep 16 00:38:46.745729 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:38:46.745697 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7df985_5a6c_451f_8d55_f88d795f0195.slice/crio-01c8960752c92b4748bbed523f632460c46182966944a92cf7d14f92dc2a20e6 WatchSource:0}: Error finding container 01c8960752c92b4748bbed523f632460c46182966944a92cf7d14f92dc2a20e6: Status 404 returned error can't find the container with id 01c8960752c92b4748bbed523f632460c46182966944a92cf7d14f92dc2a20e6 Sep 16 00:38:47.240058 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:47.240009 2567 generic.go:356] "Generic (PLEG): container finished" podID="0d7df985-5a6c-451f-8d55-f88d795f0195" containerID="e878f40c5a5333431aa2023212757dfb547a2103d395258081e6df66b8a8aafd" exitCode=0 Sep 16 00:38:47.240058 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:47.240060 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w" event={"ID":"0d7df985-5a6c-451f-8d55-f88d795f0195","Type":"ContainerDied","Data":"e878f40c5a5333431aa2023212757dfb547a2103d395258081e6df66b8a8aafd"} Sep 16 00:38:47.240461 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:47.240122 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w" event={"ID":"0d7df985-5a6c-451f-8d55-f88d795f0195","Type":"ContainerStarted","Data":"01c8960752c92b4748bbed523f632460c46182966944a92cf7d14f92dc2a20e6"} Sep 16 00:38:47.460304 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:47.460275 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:47.560730 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:47.560701 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:48.124610 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.124580 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:48.245723 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.245697 2567 generic.go:356] "Generic (PLEG): container finished" podID="0d7df985-5a6c-451f-8d55-f88d795f0195" containerID="36ecfee9f578a1a8635e340c1dd2595ea4450277b09f41b45bb4db455b81cb87" exitCode=0 Sep 16 00:38:48.245993 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.245765 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w" event={"ID":"0d7df985-5a6c-451f-8d55-f88d795f0195","Type":"ContainerDied","Data":"36ecfee9f578a1a8635e340c1dd2595ea4450277b09f41b45bb4db455b81cb87"} Sep 16 00:38:48.367730 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.367696 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:48.469366 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.469337 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:48.475487 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.475459 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q"] Sep 16 00:38:48.477641 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.477626 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" Sep 16 00:38:48.480432 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.480403 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 16 00:38:48.480432 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.480410 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 16 00:38:48.480585 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.480506 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 16 00:38:48.480585 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.480537 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zp8vf\"" Sep 16 00:38:48.483088 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.483072 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 16 00:38:48.496298 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.496279 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q"] Sep 16 00:38:48.607631 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.607607 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fv2f\" (UniqueName: \"kubernetes.io/projected/0d37134c-6aa4-4876-8992-0274c84ebd5c-kube-api-access-6fv2f\") pod \"opendatahub-operator-controller-manager-55d645f7ff-gv42q\" (UID: \"0d37134c-6aa4-4876-8992-0274c84ebd5c\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" Sep 16 00:38:48.607758 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.607642 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d37134c-6aa4-4876-8992-0274c84ebd5c-tmp\") pod \"opendatahub-operator-controller-manager-55d645f7ff-gv42q\" (UID: \"0d37134c-6aa4-4876-8992-0274c84ebd5c\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" Sep 16 00:38:48.607758 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.607670 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0d37134c-6aa4-4876-8992-0274c84ebd5c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d645f7ff-gv42q\" (UID: \"0d37134c-6aa4-4876-8992-0274c84ebd5c\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" Sep 16 00:38:48.607758 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.607693 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d37134c-6aa4-4876-8992-0274c84ebd5c-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d645f7ff-gv42q\" (UID: \"0d37134c-6aa4-4876-8992-0274c84ebd5c\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" Sep 16 00:38:48.607758 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.607753 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0d37134c-6aa4-4876-8992-0274c84ebd5c-manifests\") pod \"opendatahub-operator-controller-manager-55d645f7ff-gv42q\" (UID: \"0d37134c-6aa4-4876-8992-0274c84ebd5c\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" Sep 16 00:38:48.670671 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.670639 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:48.708703 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.708677 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0d37134c-6aa4-4876-8992-0274c84ebd5c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d645f7ff-gv42q\" (UID: \"0d37134c-6aa4-4876-8992-0274c84ebd5c\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" Sep 16 00:38:48.708802 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.708704 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d37134c-6aa4-4876-8992-0274c84ebd5c-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d645f7ff-gv42q\" (UID: \"0d37134c-6aa4-4876-8992-0274c84ebd5c\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" Sep 16 00:38:48.708802 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.708767 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0d37134c-6aa4-4876-8992-0274c84ebd5c-manifests\") pod \"opendatahub-operator-controller-manager-55d645f7ff-gv42q\" (UID: \"0d37134c-6aa4-4876-8992-0274c84ebd5c\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" Sep 16 00:38:48.708802 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.708795 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fv2f\" (UniqueName: \"kubernetes.io/projected/0d37134c-6aa4-4876-8992-0274c84ebd5c-kube-api-access-6fv2f\") pod \"opendatahub-operator-controller-manager-55d645f7ff-gv42q\" (UID: \"0d37134c-6aa4-4876-8992-0274c84ebd5c\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" Sep 16 00:38:48.708942 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.708844 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d37134c-6aa4-4876-8992-0274c84ebd5c-tmp\") pod \"opendatahub-operator-controller-manager-55d645f7ff-gv42q\" (UID: \"0d37134c-6aa4-4876-8992-0274c84ebd5c\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" Sep 16 00:38:48.709198 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.709180 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0d37134c-6aa4-4876-8992-0274c84ebd5c-manifests\") pod \"opendatahub-operator-controller-manager-55d645f7ff-gv42q\" (UID: \"0d37134c-6aa4-4876-8992-0274c84ebd5c\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" Sep 16 00:38:48.711386 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.711360 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d37134c-6aa4-4876-8992-0274c84ebd5c-tmp\") pod \"opendatahub-operator-controller-manager-55d645f7ff-gv42q\" (UID: \"0d37134c-6aa4-4876-8992-0274c84ebd5c\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" Sep 16 00:38:48.711386 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.711376 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d37134c-6aa4-4876-8992-0274c84ebd5c-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d645f7ff-gv42q\" (UID: \"0d37134c-6aa4-4876-8992-0274c84ebd5c\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" Sep 16 00:38:48.711495 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.711479 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0d37134c-6aa4-4876-8992-0274c84ebd5c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d645f7ff-gv42q\" (UID: \"0d37134c-6aa4-4876-8992-0274c84ebd5c\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" Sep 16 00:38:48.720076 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.720028 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fv2f\" (UniqueName: \"kubernetes.io/projected/0d37134c-6aa4-4876-8992-0274c84ebd5c-kube-api-access-6fv2f\") pod \"opendatahub-operator-controller-manager-55d645f7ff-gv42q\" (UID: \"0d37134c-6aa4-4876-8992-0274c84ebd5c\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" Sep 16 00:38:48.780179 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.780154 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:48.788348 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.788326 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" Sep 16 00:38:48.871377 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.871325 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:48.972003 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:48.971958 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:49.073270 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:49.073245 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q"] Sep 16 00:38:49.073856 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:49.073838 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:49.075906 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:38:49.075883 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d37134c_6aa4_4876_8992_0274c84ebd5c.slice/crio-bf23e59630d3bbb685d7144a794780ae6cc1588a37879f016b1e2a4748079bb2 WatchSource:0}: Error finding container bf23e59630d3bbb685d7144a794780ae6cc1588a37879f016b1e2a4748079bb2: Status 404 returned error can't find the container with id bf23e59630d3bbb685d7144a794780ae6cc1588a37879f016b1e2a4748079bb2 Sep 16 00:38:49.171615 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:49.171536 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:38:49.252076 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:49.252023 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" event={"ID":"0d37134c-6aa4-4876-8992-0274c84ebd5c","Type":"ContainerStarted","Data":"bf23e59630d3bbb685d7144a794780ae6cc1588a37879f016b1e2a4748079bb2"} Sep 16 00:38:49.254306 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:49.254286 2567 generic.go:356] "Generic (PLEG): container finished" podID="0d7df985-5a6c-451f-8d55-f88d795f0195" containerID="34f1da4abf48da57663c79be702071fb8a39f803ea2696911ad3ea7666ad4146" exitCode=0 Sep 16 00:38:49.254388 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:49.254337 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w" event={"ID":"0d7df985-5a6c-451f-8d55-f88d795f0195","Type":"ContainerDied","Data":"34f1da4abf48da57663c79be702071fb8a39f803ea2696911ad3ea7666ad4146"} Sep 16 00:38:50.397392 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:50.397366 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w" Sep 16 00:38:50.522813 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:50.522751 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7rkl\" (UniqueName: \"kubernetes.io/projected/0d7df985-5a6c-451f-8d55-f88d795f0195-kube-api-access-d7rkl\") pod \"0d7df985-5a6c-451f-8d55-f88d795f0195\" (UID: \"0d7df985-5a6c-451f-8d55-f88d795f0195\") " Sep 16 00:38:50.522813 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:50.522798 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d7df985-5a6c-451f-8d55-f88d795f0195-util\") pod \"0d7df985-5a6c-451f-8d55-f88d795f0195\" (UID: \"0d7df985-5a6c-451f-8d55-f88d795f0195\") " Sep 16 00:38:50.522994 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:50.522829 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d7df985-5a6c-451f-8d55-f88d795f0195-bundle\") pod \"0d7df985-5a6c-451f-8d55-f88d795f0195\" (UID: \"0d7df985-5a6c-451f-8d55-f88d795f0195\") " Sep 16 00:38:50.523704 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:50.523640 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7df985-5a6c-451f-8d55-f88d795f0195-bundle" (OuterVolumeSpecName: "bundle") pod "0d7df985-5a6c-451f-8d55-f88d795f0195" (UID: "0d7df985-5a6c-451f-8d55-f88d795f0195"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:38:50.525701 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:50.525670 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7df985-5a6c-451f-8d55-f88d795f0195-kube-api-access-d7rkl" (OuterVolumeSpecName: "kube-api-access-d7rkl") pod "0d7df985-5a6c-451f-8d55-f88d795f0195" (UID: "0d7df985-5a6c-451f-8d55-f88d795f0195"). InnerVolumeSpecName "kube-api-access-d7rkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:38:50.530727 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:50.530669 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7df985-5a6c-451f-8d55-f88d795f0195-util" (OuterVolumeSpecName: "util") pod "0d7df985-5a6c-451f-8d55-f88d795f0195" (UID: "0d7df985-5a6c-451f-8d55-f88d795f0195"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:38:50.624291 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:50.624231 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7rkl\" (UniqueName: \"kubernetes.io/projected/0d7df985-5a6c-451f-8d55-f88d795f0195-kube-api-access-d7rkl\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:38:50.624291 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:50.624261 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d7df985-5a6c-451f-8d55-f88d795f0195-util\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:38:50.624291 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:50.624272 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d7df985-5a6c-451f-8d55-f88d795f0195-bundle\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:38:51.267760 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:51.267721 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w" event={"ID":"0d7df985-5a6c-451f-8d55-f88d795f0195","Type":"ContainerDied","Data":"01c8960752c92b4748bbed523f632460c46182966944a92cf7d14f92dc2a20e6"} Sep 16 00:38:51.267937 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:51.267780 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01c8960752c92b4748bbed523f632460c46182966944a92cf7d14f92dc2a20e6" Sep 16 00:38:51.267937 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:51.267737 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07cb7w" Sep 16 00:38:52.273750 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:52.273682 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" event={"ID":"0d37134c-6aa4-4876-8992-0274c84ebd5c","Type":"ContainerStarted","Data":"02497fbb855ee43d351a16af7f2474b21399ce9cb1f7832ce1850ae484de78e1"} Sep 16 00:38:53.279765 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:53.279732 2567 generic.go:356] "Generic (PLEG): container finished" podID="0d37134c-6aa4-4876-8992-0274c84ebd5c" containerID="02497fbb855ee43d351a16af7f2474b21399ce9cb1f7832ce1850ae484de78e1" exitCode=0 Sep 16 00:38:53.280120 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:53.279787 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" event={"ID":"0d37134c-6aa4-4876-8992-0274c84ebd5c","Type":"ContainerDied","Data":"02497fbb855ee43d351a16af7f2474b21399ce9cb1f7832ce1850ae484de78e1"} Sep 16 00:38:54.286350 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:54.286311 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" event={"ID":"0d37134c-6aa4-4876-8992-0274c84ebd5c","Type":"ContainerStarted","Data":"436de7ff241b882c77859a88d43de1813e26345b1e8288110c4b197f673be127"} Sep 16 00:38:54.286769 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:54.286441 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" Sep 16 00:38:54.309811 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:54.309771 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" podStartSLOduration=1.9595940779999999 podStartE2EDuration="6.309754795s" totalImagesPullingTime="3.27830635s" totalInitContainerRuntime="1.071854367s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:38:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:38:54.308516222 +0000 UTC m=+284.670739753" watchObservedRunningTime="2026-09-16 00:38:54.309754795 +0000 UTC m=+284.671978326" Sep 16 00:38:58.917972 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:38:58.917940 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:00.243338 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:00.243292 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:05.292744 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:05.292707 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-gv42q" Sep 16 00:39:06.137233 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.137187 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:06.216712 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.216677 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:06.217352 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.217321 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-864798458c-v8zs5"] Sep 16 00:39:06.217637 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.217623 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d7df985-5a6c-451f-8d55-f88d795f0195" containerName="util" Sep 16 00:39:06.217714 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.217640 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d7df985-5a6c-451f-8d55-f88d795f0195" containerName="util" Sep 16 00:39:06.217714 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.217650 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d7df985-5a6c-451f-8d55-f88d795f0195" containerName="extract" Sep 16 00:39:06.217714 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.217657 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d7df985-5a6c-451f-8d55-f88d795f0195" containerName="extract" Sep 16 00:39:06.217714 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.217671 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d7df985-5a6c-451f-8d55-f88d795f0195" containerName="pull" Sep 16 00:39:06.217714 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.217680 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d7df985-5a6c-451f-8d55-f88d795f0195" containerName="pull" Sep 16 00:39:06.217954 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.217767 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d7df985-5a6c-451f-8d55-f88d795f0195" containerName="extract" Sep 16 00:39:06.221396 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.221375 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-864798458c-v8zs5" Sep 16 00:39:06.225379 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.225359 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 16 00:39:06.225516 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.225387 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 16 00:39:06.225516 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.225391 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cbpxz\"" Sep 16 00:39:06.225516 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.225429 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 16 00:39:06.237400 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.237379 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-864798458c-v8zs5"] Sep 16 00:39:06.341589 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.341561 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5387ab5d-9440-4c3c-88ea-3d2188cbdbed-cert\") pod \"lws-controller-manager-864798458c-v8zs5\" (UID: \"5387ab5d-9440-4c3c-88ea-3d2188cbdbed\") " pod="openshift-lws-operator/lws-controller-manager-864798458c-v8zs5" Sep 16 00:39:06.341589 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.341588 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5387ab5d-9440-4c3c-88ea-3d2188cbdbed-metrics-cert\") pod \"lws-controller-manager-864798458c-v8zs5\" (UID: \"5387ab5d-9440-4c3c-88ea-3d2188cbdbed\") " pod="openshift-lws-operator/lws-controller-manager-864798458c-v8zs5" Sep 16 00:39:06.341964 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.341604 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5387ab5d-9440-4c3c-88ea-3d2188cbdbed-manager-config\") pod \"lws-controller-manager-864798458c-v8zs5\" (UID: \"5387ab5d-9440-4c3c-88ea-3d2188cbdbed\") " pod="openshift-lws-operator/lws-controller-manager-864798458c-v8zs5" Sep 16 00:39:06.341964 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.341626 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5cpl\" (UniqueName: \"kubernetes.io/projected/5387ab5d-9440-4c3c-88ea-3d2188cbdbed-kube-api-access-g5cpl\") pod \"lws-controller-manager-864798458c-v8zs5\" (UID: \"5387ab5d-9440-4c3c-88ea-3d2188cbdbed\") " pod="openshift-lws-operator/lws-controller-manager-864798458c-v8zs5" Sep 16 00:39:06.442445 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.442380 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5387ab5d-9440-4c3c-88ea-3d2188cbdbed-cert\") pod \"lws-controller-manager-864798458c-v8zs5\" (UID: \"5387ab5d-9440-4c3c-88ea-3d2188cbdbed\") " pod="openshift-lws-operator/lws-controller-manager-864798458c-v8zs5" Sep 16 00:39:06.442445 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.442416 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5387ab5d-9440-4c3c-88ea-3d2188cbdbed-metrics-cert\") pod \"lws-controller-manager-864798458c-v8zs5\" (UID: \"5387ab5d-9440-4c3c-88ea-3d2188cbdbed\") " pod="openshift-lws-operator/lws-controller-manager-864798458c-v8zs5" Sep 16 00:39:06.442610 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.442538 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5387ab5d-9440-4c3c-88ea-3d2188cbdbed-manager-config\") pod \"lws-controller-manager-864798458c-v8zs5\" (UID: \"5387ab5d-9440-4c3c-88ea-3d2188cbdbed\") " pod="openshift-lws-operator/lws-controller-manager-864798458c-v8zs5" Sep 16 00:39:06.442667 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.442616 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5cpl\" (UniqueName: \"kubernetes.io/projected/5387ab5d-9440-4c3c-88ea-3d2188cbdbed-kube-api-access-g5cpl\") pod \"lws-controller-manager-864798458c-v8zs5\" (UID: \"5387ab5d-9440-4c3c-88ea-3d2188cbdbed\") " pod="openshift-lws-operator/lws-controller-manager-864798458c-v8zs5" Sep 16 00:39:06.443524 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.443496 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5387ab5d-9440-4c3c-88ea-3d2188cbdbed-manager-config\") pod \"lws-controller-manager-864798458c-v8zs5\" (UID: \"5387ab5d-9440-4c3c-88ea-3d2188cbdbed\") " pod="openshift-lws-operator/lws-controller-manager-864798458c-v8zs5" Sep 16 00:39:06.445091 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.445063 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5387ab5d-9440-4c3c-88ea-3d2188cbdbed-metrics-cert\") pod \"lws-controller-manager-864798458c-v8zs5\" (UID: \"5387ab5d-9440-4c3c-88ea-3d2188cbdbed\") " pod="openshift-lws-operator/lws-controller-manager-864798458c-v8zs5" Sep 16 00:39:06.445213 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.445192 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5387ab5d-9440-4c3c-88ea-3d2188cbdbed-cert\") pod \"lws-controller-manager-864798458c-v8zs5\" (UID: \"5387ab5d-9440-4c3c-88ea-3d2188cbdbed\") " pod="openshift-lws-operator/lws-controller-manager-864798458c-v8zs5" Sep 16 00:39:06.454556 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.454533 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5cpl\" (UniqueName: \"kubernetes.io/projected/5387ab5d-9440-4c3c-88ea-3d2188cbdbed-kube-api-access-g5cpl\") pod \"lws-controller-manager-864798458c-v8zs5\" (UID: \"5387ab5d-9440-4c3c-88ea-3d2188cbdbed\") " pod="openshift-lws-operator/lws-controller-manager-864798458c-v8zs5" Sep 16 00:39:06.530580 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.530561 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-864798458c-v8zs5" Sep 16 00:39:06.804906 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:06.804884 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-864798458c-v8zs5"] Sep 16 00:39:06.807183 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:39:06.807151 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5387ab5d_9440_4c3c_88ea_3d2188cbdbed.slice/crio-30ec930979067ae6286c3519c42236152b2c088ac14aba2969812016c9e96a72 WatchSource:0}: Error finding container 30ec930979067ae6286c3519c42236152b2c088ac14aba2969812016c9e96a72: Status 404 returned error can't find the container with id 30ec930979067ae6286c3519c42236152b2c088ac14aba2969812016c9e96a72 Sep 16 00:39:07.342464 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:07.342430 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-864798458c-v8zs5" event={"ID":"5387ab5d-9440-4c3c-88ea-3d2188cbdbed","Type":"ContainerStarted","Data":"30ec930979067ae6286c3519c42236152b2c088ac14aba2969812016c9e96a72"} Sep 16 00:39:08.032732 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:08.032700 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:08.145378 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:08.145338 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:08.497356 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:08.497314 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:08.595339 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:08.595305 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:09.354282 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:09.354246 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-864798458c-v8zs5" event={"ID":"5387ab5d-9440-4c3c-88ea-3d2188cbdbed","Type":"ContainerStarted","Data":"8598fa91c19ea803c503579a45865873fccfd40d2a5d5c23e241f0500abaa151"} Sep 16 00:39:09.354518 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:09.354426 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-864798458c-v8zs5" Sep 16 00:39:09.378911 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:09.378866 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-864798458c-v8zs5" podStartSLOduration=1.589824411 podStartE2EDuration="3.378853042s" totalImagesPullingTime="1.789028631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:39:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:39:09.376111039 +0000 UTC m=+299.738334592" watchObservedRunningTime="2026-09-16 00:39:09.378853042 +0000 UTC m=+299.741076574" Sep 16 00:39:09.690702 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:09.690627 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:09.756385 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:09.756348 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:10.004650 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:10.004582 2567 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 00:39:10.701515 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:10.701478 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:14.769182 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:14.769134 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:14.868000 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:14.867945 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:15.324561 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.324530 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8"] Sep 16 00:39:15.328198 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.328178 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.331074 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.331049 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 16 00:39:15.331074 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.331068 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-tvx7s\"" Sep 16 00:39:15.331253 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.331061 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 16 00:39:15.331253 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.331121 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 16 00:39:15.339243 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.339220 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8"] Sep 16 00:39:15.352672 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.352653 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:15.419317 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.419297 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c87bb77-2bef-4b57-80f0-b77131f2425d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.419414 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.419322 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c87bb77-2bef-4b57-80f0-b77131f2425d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.419414 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.419346 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c87bb77-2bef-4b57-80f0-b77131f2425d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.419414 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.419368 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c87bb77-2bef-4b57-80f0-b77131f2425d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.419520 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.419441 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c87bb77-2bef-4b57-80f0-b77131f2425d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.419520 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.419470 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c87bb77-2bef-4b57-80f0-b77131f2425d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.419520 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.419501 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8twz\" (UniqueName: \"kubernetes.io/projected/3c87bb77-2bef-4b57-80f0-b77131f2425d-kube-api-access-b8twz\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.419612 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.419540 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c87bb77-2bef-4b57-80f0-b77131f2425d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.419612 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.419569 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c87bb77-2bef-4b57-80f0-b77131f2425d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.520591 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.520564 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c87bb77-2bef-4b57-80f0-b77131f2425d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.520591 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.520594 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c87bb77-2bef-4b57-80f0-b77131f2425d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.520825 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.520619 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c87bb77-2bef-4b57-80f0-b77131f2425d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.520825 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.520641 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c87bb77-2bef-4b57-80f0-b77131f2425d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.520825 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.520666 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c87bb77-2bef-4b57-80f0-b77131f2425d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.520825 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.520691 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c87bb77-2bef-4b57-80f0-b77131f2425d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.520825 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.520734 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8twz\" (UniqueName: \"kubernetes.io/projected/3c87bb77-2bef-4b57-80f0-b77131f2425d-kube-api-access-b8twz\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.520825 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.520789 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c87bb77-2bef-4b57-80f0-b77131f2425d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.521099 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.520933 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c87bb77-2bef-4b57-80f0-b77131f2425d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.521539 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.521474 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c87bb77-2bef-4b57-80f0-b77131f2425d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.521672 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.521537 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c87bb77-2bef-4b57-80f0-b77131f2425d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.521672 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.521542 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c87bb77-2bef-4b57-80f0-b77131f2425d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.521770 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.521710 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c87bb77-2bef-4b57-80f0-b77131f2425d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.522006 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.521990 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c87bb77-2bef-4b57-80f0-b77131f2425d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.523845 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.523830 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c87bb77-2bef-4b57-80f0-b77131f2425d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.524193 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.524173 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c87bb77-2bef-4b57-80f0-b77131f2425d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.530381 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.530356 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c87bb77-2bef-4b57-80f0-b77131f2425d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.530851 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.530828 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8twz\" (UniqueName: \"kubernetes.io/projected/3c87bb77-2bef-4b57-80f0-b77131f2425d-kube-api-access-b8twz\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8\" (UID: \"3c87bb77-2bef-4b57-80f0-b77131f2425d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.544991 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.544967 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:15.639745 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.639694 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:15.916497 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.916471 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8"] Sep 16 00:39:15.918815 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:39:15.918787 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c87bb77_2bef_4b57_80f0_b77131f2425d.slice/crio-314ebbd5c58fbe0574932eba713b45b783738c2bd71bc96d3b9b32fc285c4b0d WatchSource:0}: Error finding container 314ebbd5c58fbe0574932eba713b45b783738c2bd71bc96d3b9b32fc285c4b0d: Status 404 returned error can't find the container with id 314ebbd5c58fbe0574932eba713b45b783738c2bd71bc96d3b9b32fc285c4b0d Sep 16 00:39:15.921062 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.921016 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 00:39:15.970032 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:15.970004 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:16.384209 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:16.384182 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" event={"ID":"3c87bb77-2bef-4b57-80f0-b77131f2425d","Type":"ContainerStarted","Data":"314ebbd5c58fbe0574932eba713b45b783738c2bd71bc96d3b9b32fc285c4b0d"} Sep 16 00:39:18.779597 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:18.779550 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:39:18.779887 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:18.779639 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:39:18.779887 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:18.779669 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:39:19.097781 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.097751 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lgdjj"] Sep 16 00:39:19.101077 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.101028 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lgdjj" Sep 16 00:39:19.103945 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.103920 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-jctcm\"" Sep 16 00:39:19.104137 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.104117 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 16 00:39:19.110619 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.110597 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lgdjj"] Sep 16 00:39:19.127095 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.127064 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:19.227176 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.227150 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:19.252596 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.252555 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdqvb\" (UniqueName: \"kubernetes.io/projected/0a62d9e3-188b-41ef-96fa-47ac89811ce9-kube-api-access-zdqvb\") pod \"odh-model-controller-7998c67dc4-lgdjj\" (UID: \"0a62d9e3-188b-41ef-96fa-47ac89811ce9\") " pod="opendatahub/odh-model-controller-7998c67dc4-lgdjj" Sep 16 00:39:19.252765 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.252707 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a62d9e3-188b-41ef-96fa-47ac89811ce9-cert\") pod \"odh-model-controller-7998c67dc4-lgdjj\" (UID: \"0a62d9e3-188b-41ef-96fa-47ac89811ce9\") " pod="opendatahub/odh-model-controller-7998c67dc4-lgdjj" Sep 16 00:39:19.354229 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.354145 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdqvb\" (UniqueName: \"kubernetes.io/projected/0a62d9e3-188b-41ef-96fa-47ac89811ce9-kube-api-access-zdqvb\") pod \"odh-model-controller-7998c67dc4-lgdjj\" (UID: \"0a62d9e3-188b-41ef-96fa-47ac89811ce9\") " pod="opendatahub/odh-model-controller-7998c67dc4-lgdjj" Sep 16 00:39:19.354229 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.354213 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a62d9e3-188b-41ef-96fa-47ac89811ce9-cert\") pod \"odh-model-controller-7998c67dc4-lgdjj\" (UID: \"0a62d9e3-188b-41ef-96fa-47ac89811ce9\") " pod="opendatahub/odh-model-controller-7998c67dc4-lgdjj" Sep 16 00:39:19.354466 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:39:19.354313 2567 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 16 00:39:19.354466 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:39:19.354367 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a62d9e3-188b-41ef-96fa-47ac89811ce9-cert podName:0a62d9e3-188b-41ef-96fa-47ac89811ce9 nodeName:}" failed. No retries permitted until 2026-09-16 00:39:19.854350039 +0000 UTC m=+310.216573549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0a62d9e3-188b-41ef-96fa-47ac89811ce9-cert") pod "odh-model-controller-7998c67dc4-lgdjj" (UID: "0a62d9e3-188b-41ef-96fa-47ac89811ce9") : secret "odh-model-controller-webhook-cert" not found Sep 16 00:39:19.363738 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.363688 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdqvb\" (UniqueName: \"kubernetes.io/projected/0a62d9e3-188b-41ef-96fa-47ac89811ce9-kube-api-access-zdqvb\") pod \"odh-model-controller-7998c67dc4-lgdjj\" (UID: \"0a62d9e3-188b-41ef-96fa-47ac89811ce9\") " pod="opendatahub/odh-model-controller-7998c67dc4-lgdjj" Sep 16 00:39:19.401575 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.401538 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" event={"ID":"3c87bb77-2bef-4b57-80f0-b77131f2425d","Type":"ContainerStarted","Data":"f9dd4f4db904cd2e493cb871cc24051958c19bc8b4dc88596f7549c2cccabfdf"} Sep 16 00:39:19.421613 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.421574 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" podStartSLOduration=1.5635597909999999 podStartE2EDuration="4.42156069s" totalImagesPullingTime="2.858000899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:39:19.419857088 +0000 UTC m=+309.782080634" watchObservedRunningTime="2026-09-16 00:39:19.42156069 +0000 UTC m=+309.783784221" Sep 16 00:39:19.534379 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.534342 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-d77jm"] Sep 16 00:39:19.537633 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.537612 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-d77jm" Sep 16 00:39:19.539757 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.539731 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:19.540178 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.540162 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 16 00:39:19.548066 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.548028 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-d77jm"] Sep 16 00:39:19.636890 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.636810 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:19.639887 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.639855 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:19.641406 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.641380 2567 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Sep 16 00:39:19.641493 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.641446 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" podUID="3c87bb77-2bef-4b57-80f0-b77131f2425d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Sep 16 00:39:19.657972 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.657944 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sbcw\" (UniqueName: \"kubernetes.io/projected/074346fa-1298-4edd-b974-baf07217275a-kube-api-access-9sbcw\") pod \"model-serving-api-54d89bd5fc-d77jm\" (UID: \"074346fa-1298-4edd-b974-baf07217275a\") " pod="opendatahub/model-serving-api-54d89bd5fc-d77jm" Sep 16 00:39:19.658152 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.658087 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/074346fa-1298-4edd-b974-baf07217275a-tls-certs\") pod \"model-serving-api-54d89bd5fc-d77jm\" (UID: \"074346fa-1298-4edd-b974-baf07217275a\") " pod="opendatahub/model-serving-api-54d89bd5fc-d77jm" Sep 16 00:39:19.759319 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.759283 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sbcw\" (UniqueName: \"kubernetes.io/projected/074346fa-1298-4edd-b974-baf07217275a-kube-api-access-9sbcw\") pod \"model-serving-api-54d89bd5fc-d77jm\" (UID: \"074346fa-1298-4edd-b974-baf07217275a\") " pod="opendatahub/model-serving-api-54d89bd5fc-d77jm" Sep 16 00:39:19.759519 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.759384 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/074346fa-1298-4edd-b974-baf07217275a-tls-certs\") pod \"model-serving-api-54d89bd5fc-d77jm\" (UID: \"074346fa-1298-4edd-b974-baf07217275a\") " pod="opendatahub/model-serving-api-54d89bd5fc-d77jm" Sep 16 00:39:19.759519 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:39:19.759489 2567 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 16 00:39:19.759632 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:39:19.759539 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074346fa-1298-4edd-b974-baf07217275a-tls-certs podName:074346fa-1298-4edd-b974-baf07217275a nodeName:}" failed. No retries permitted until 2026-09-16 00:39:20.259526202 +0000 UTC m=+310.621749712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/074346fa-1298-4edd-b974-baf07217275a-tls-certs") pod "model-serving-api-54d89bd5fc-d77jm" (UID: "074346fa-1298-4edd-b974-baf07217275a") : secret "model-serving-api-tls" not found Sep 16 00:39:19.768190 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.768162 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sbcw\" (UniqueName: \"kubernetes.io/projected/074346fa-1298-4edd-b974-baf07217275a-kube-api-access-9sbcw\") pod \"model-serving-api-54d89bd5fc-d77jm\" (UID: \"074346fa-1298-4edd-b974-baf07217275a\") " pod="opendatahub/model-serving-api-54d89bd5fc-d77jm" Sep 16 00:39:19.860365 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.860323 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a62d9e3-188b-41ef-96fa-47ac89811ce9-cert\") pod \"odh-model-controller-7998c67dc4-lgdjj\" (UID: \"0a62d9e3-188b-41ef-96fa-47ac89811ce9\") " pod="opendatahub/odh-model-controller-7998c67dc4-lgdjj" Sep 16 00:39:19.863393 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:19.863363 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a62d9e3-188b-41ef-96fa-47ac89811ce9-cert\") pod \"odh-model-controller-7998c67dc4-lgdjj\" (UID: \"0a62d9e3-188b-41ef-96fa-47ac89811ce9\") " pod="opendatahub/odh-model-controller-7998c67dc4-lgdjj" Sep 16 00:39:20.015000 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:20.014926 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lgdjj" Sep 16 00:39:20.061565 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:20.061507 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:20.263943 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:20.263905 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/074346fa-1298-4edd-b974-baf07217275a-tls-certs\") pod \"model-serving-api-54d89bd5fc-d77jm\" (UID: \"074346fa-1298-4edd-b974-baf07217275a\") " pod="opendatahub/model-serving-api-54d89bd5fc-d77jm" Sep 16 00:39:20.266978 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:20.266895 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/074346fa-1298-4edd-b974-baf07217275a-tls-certs\") pod \"model-serving-api-54d89bd5fc-d77jm\" (UID: \"074346fa-1298-4edd-b974-baf07217275a\") " pod="opendatahub/model-serving-api-54d89bd5fc-d77jm" Sep 16 00:39:20.301719 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:20.301665 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lgdjj"] Sep 16 00:39:20.303902 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:39:20.303872 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a62d9e3_188b_41ef_96fa_47ac89811ce9.slice/crio-8e16c7c642bcc96d5b6e6646ab110ada53d2757648555e02f6838f742efc9fe3 WatchSource:0}: Error finding container 8e16c7c642bcc96d5b6e6646ab110ada53d2757648555e02f6838f742efc9fe3: Status 404 returned error can't find the container with id 8e16c7c642bcc96d5b6e6646ab110ada53d2757648555e02f6838f742efc9fe3 Sep 16 00:39:20.360861 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:20.360834 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-864798458c-v8zs5" Sep 16 00:39:20.408759 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:20.408721 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lgdjj" event={"ID":"0a62d9e3-188b-41ef-96fa-47ac89811ce9","Type":"ContainerStarted","Data":"8e16c7c642bcc96d5b6e6646ab110ada53d2757648555e02f6838f742efc9fe3"} Sep 16 00:39:20.450192 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:20.450151 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-d77jm" Sep 16 00:39:20.644700 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:20.644672 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:20.723209 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:20.723187 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-d77jm"] Sep 16 00:39:20.725630 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:39:20.725603 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod074346fa_1298_4edd_b974_baf07217275a.slice/crio-64ab196925ef7ee9393afca0dcb8201d4b7756716fe16431485da94137126d84 WatchSource:0}: Error finding container 64ab196925ef7ee9393afca0dcb8201d4b7756716fe16431485da94137126d84: Status 404 returned error can't find the container with id 64ab196925ef7ee9393afca0dcb8201d4b7756716fe16431485da94137126d84 Sep 16 00:39:21.417678 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:21.417641 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-d77jm" event={"ID":"074346fa-1298-4edd-b974-baf07217275a","Type":"ContainerStarted","Data":"64ab196925ef7ee9393afca0dcb8201d4b7756716fe16431485da94137126d84"} Sep 16 00:39:21.418241 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:21.418192 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:21.419305 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:21.419248 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8" Sep 16 00:39:25.444979 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:25.444942 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lgdjj" event={"ID":"0a62d9e3-188b-41ef-96fa-47ac89811ce9","Type":"ContainerStarted","Data":"8d1370e0e46aa87399c0ed8a6bdef12c3ba845c032391d80615b78a3f1f3eeb4"} Sep 16 00:39:25.445566 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:25.445097 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-lgdjj" Sep 16 00:39:25.446748 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:25.446717 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-d77jm" event={"ID":"074346fa-1298-4edd-b974-baf07217275a","Type":"ContainerStarted","Data":"9754deac2514b7348406f9050acd398b9b4de117833e560be266511448e74920"} Sep 16 00:39:25.446880 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:25.446802 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-d77jm" Sep 16 00:39:25.462228 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:25.462187 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-lgdjj" podStartSLOduration=2.0972465 podStartE2EDuration="6.462157141s" totalImagesPullingTime="4.364910641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:39:25.460345376 +0000 UTC m=+315.822568932" watchObservedRunningTime="2026-09-16 00:39:25.462157141 +0000 UTC m=+315.824380672" Sep 16 00:39:25.476602 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:25.476188 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-d77jm" podStartSLOduration=2.494337561 podStartE2EDuration="6.476169287s" totalImagesPullingTime="3.981831726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:39:25.475456589 +0000 UTC m=+315.837680139" watchObservedRunningTime="2026-09-16 00:39:25.476169287 +0000 UTC m=+315.838392817" Sep 16 00:39:26.967094 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:26.967061 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk"] Sep 16 00:39:26.970602 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:26.970583 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:26.973291 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:26.973268 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 16 00:39:26.973291 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:26.973286 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 16 00:39:26.973449 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:26.973286 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 16 00:39:26.973449 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:26.973330 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vcsn2\"" Sep 16 00:39:26.982484 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:26.982461 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk"] Sep 16 00:39:27.028194 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.028164 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8acf6a17-0628-4c35-bf78-c3dc03adecd8-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.028304 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.028200 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8acf6a17-0628-4c35-bf78-c3dc03adecd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.028304 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.028241 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8acf6a17-0628-4c35-bf78-c3dc03adecd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.028304 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.028292 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8acf6a17-0628-4c35-bf78-c3dc03adecd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.028445 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.028341 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xb9l\" (UniqueName: \"kubernetes.io/projected/8acf6a17-0628-4c35-bf78-c3dc03adecd8-kube-api-access-4xb9l\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.028445 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.028386 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8acf6a17-0628-4c35-bf78-c3dc03adecd8-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.028445 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.028427 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acf6a17-0628-4c35-bf78-c3dc03adecd8-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.129367 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.129344 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8acf6a17-0628-4c35-bf78-c3dc03adecd8-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.129472 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.129390 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acf6a17-0628-4c35-bf78-c3dc03adecd8-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.129560 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.129541 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8acf6a17-0628-4c35-bf78-c3dc03adecd8-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.129618 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.129572 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8acf6a17-0628-4c35-bf78-c3dc03adecd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.129934 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.129913 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8acf6a17-0628-4c35-bf78-c3dc03adecd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.130019 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.129954 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8acf6a17-0628-4c35-bf78-c3dc03adecd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.130019 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.129984 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xb9l\" (UniqueName: \"kubernetes.io/projected/8acf6a17-0628-4c35-bf78-c3dc03adecd8-kube-api-access-4xb9l\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.130848 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.130821 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8acf6a17-0628-4c35-bf78-c3dc03adecd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.131825 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.131794 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acf6a17-0628-4c35-bf78-c3dc03adecd8-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.132094 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.132072 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8acf6a17-0628-4c35-bf78-c3dc03adecd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.132179 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.132100 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8acf6a17-0628-4c35-bf78-c3dc03adecd8-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.132335 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.132313 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8acf6a17-0628-4c35-bf78-c3dc03adecd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.138271 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.138250 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8acf6a17-0628-4c35-bf78-c3dc03adecd8-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.138342 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.138279 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xb9l\" (UniqueName: \"kubernetes.io/projected/8acf6a17-0628-4c35-bf78-c3dc03adecd8-kube-api-access-4xb9l\") pod \"istiod-openshift-gateway-5677ccb49b-kt8vk\" (UID: \"8acf6a17-0628-4c35-bf78-c3dc03adecd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.279865 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.279812 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:27.560021 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:27.559901 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk"] Sep 16 00:39:27.562273 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:39:27.562243 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8acf6a17_0628_4c35_bf78_c3dc03adecd8.slice/crio-dfff89efede6abd4eda9080c91b8af6f1c380326b3348758cfe38114518b784b WatchSource:0}: Error finding container dfff89efede6abd4eda9080c91b8af6f1c380326b3348758cfe38114518b784b: Status 404 returned error can't find the container with id dfff89efede6abd4eda9080c91b8af6f1c380326b3348758cfe38114518b784b Sep 16 00:39:28.464305 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:28.464266 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" event={"ID":"8acf6a17-0628-4c35-bf78-c3dc03adecd8","Type":"ContainerStarted","Data":"dfff89efede6abd4eda9080c91b8af6f1c380326b3348758cfe38114518b784b"} Sep 16 00:39:29.481314 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:29.481240 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:29.554836 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:29.554808 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:29.784296 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:29.784187 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:29.902116 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:29.902082 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:39:29.902216 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:29.902147 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:39:29.903188 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:29.903161 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:30.006396 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:30.006368 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:39:30.476922 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:30.476884 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" event={"ID":"8acf6a17-0628-4c35-bf78-c3dc03adecd8","Type":"ContainerStarted","Data":"370db270243f3ccb85ac17c1821ca74ca380cfe6354372fd42bc2be76ec0c75d"} Sep 16 00:39:30.477108 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:30.477023 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:30.478493 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:30.478470 2567 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-kt8vk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 00:39:30.478613 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:30.478513 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" podUID="8acf6a17-0628-4c35-bf78-c3dc03adecd8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 00:39:30.501891 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:30.501840 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" podStartSLOduration=2.164369028 podStartE2EDuration="4.501827989s" totalImagesPullingTime="2.337458961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:39:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:39:30.501812116 +0000 UTC m=+320.864035666" watchObservedRunningTime="2026-09-16 00:39:30.501827989 +0000 UTC m=+320.864051520" Sep 16 00:39:31.482391 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:31.482362 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kt8vk" Sep 16 00:39:36.454586 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:36.454557 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-lgdjj" Sep 16 00:39:36.458088 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:39:36.458064 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-d77jm" Sep 16 00:40:05.256759 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:05.256663 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:07.284357 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:07.284319 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:07.423138 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:07.423106 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:08.019346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:08.019311 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:08.068838 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:08.068813 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:09.328862 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:09.328835 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:09.429653 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:09.429620 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:11.282198 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:11.282151 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:18.318983 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:18.318949 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ff79b7c9f-qhnnv"] Sep 16 00:40:18.420130 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:18.420098 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:18.469095 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:18.469066 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:20.243529 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:20.243486 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:20.341096 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:20.341065 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:23.889509 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:23.889469 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm"] Sep 16 00:40:23.896845 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:23.896816 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm" Sep 16 00:40:23.900559 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:23.900533 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 00:40:23.900795 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:23.900777 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 00:40:23.900912 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:23.900862 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gw647\"" Sep 16 00:40:23.902190 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:23.902166 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm"] Sep 16 00:40:23.979684 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:23.979658 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d20df808-8e01-429a-8a27-cb9ee653c2e3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm\" (UID: \"d20df808-8e01-429a-8a27-cb9ee653c2e3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm" Sep 16 00:40:23.979806 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:23.979738 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d20df808-8e01-429a-8a27-cb9ee653c2e3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm\" (UID: \"d20df808-8e01-429a-8a27-cb9ee653c2e3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm" Sep 16 00:40:23.979806 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:23.979764 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvrjc\" (UniqueName: \"kubernetes.io/projected/d20df808-8e01-429a-8a27-cb9ee653c2e3-kube-api-access-rvrjc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm\" (UID: \"d20df808-8e01-429a-8a27-cb9ee653c2e3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm" Sep 16 00:40:23.992263 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:23.992238 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf"] Sep 16 00:40:23.995567 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:23.995553 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf" Sep 16 00:40:24.002835 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.002814 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf"] Sep 16 00:40:24.081008 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.080982 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d20df808-8e01-429a-8a27-cb9ee653c2e3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm\" (UID: \"d20df808-8e01-429a-8a27-cb9ee653c2e3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm" Sep 16 00:40:24.081169 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.081015 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c801be2-5b65-4113-9e33-764187508cc8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf\" (UID: \"1c801be2-5b65-4113-9e33-764187508cc8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf" Sep 16 00:40:24.081169 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.081069 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvrjc\" (UniqueName: \"kubernetes.io/projected/d20df808-8e01-429a-8a27-cb9ee653c2e3-kube-api-access-rvrjc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm\" (UID: \"d20df808-8e01-429a-8a27-cb9ee653c2e3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm" Sep 16 00:40:24.081169 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.081146 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d20df808-8e01-429a-8a27-cb9ee653c2e3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm\" (UID: \"d20df808-8e01-429a-8a27-cb9ee653c2e3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm" Sep 16 00:40:24.081336 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.081182 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7b9f\" (UniqueName: \"kubernetes.io/projected/1c801be2-5b65-4113-9e33-764187508cc8-kube-api-access-k7b9f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf\" (UID: \"1c801be2-5b65-4113-9e33-764187508cc8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf" Sep 16 00:40:24.081336 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.081243 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c801be2-5b65-4113-9e33-764187508cc8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf\" (UID: \"1c801be2-5b65-4113-9e33-764187508cc8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf" Sep 16 00:40:24.081407 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.081368 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d20df808-8e01-429a-8a27-cb9ee653c2e3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm\" (UID: \"d20df808-8e01-429a-8a27-cb9ee653c2e3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm" Sep 16 00:40:24.081512 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.081496 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d20df808-8e01-429a-8a27-cb9ee653c2e3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm\" (UID: \"d20df808-8e01-429a-8a27-cb9ee653c2e3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm" Sep 16 00:40:24.087789 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.087767 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2"] Sep 16 00:40:24.093494 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.093474 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2" Sep 16 00:40:24.095961 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.095937 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvrjc\" (UniqueName: \"kubernetes.io/projected/d20df808-8e01-429a-8a27-cb9ee653c2e3-kube-api-access-rvrjc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm\" (UID: \"d20df808-8e01-429a-8a27-cb9ee653c2e3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm" Sep 16 00:40:24.098751 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.098729 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2"] Sep 16 00:40:24.182165 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.182102 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c801be2-5b65-4113-9e33-764187508cc8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf\" (UID: \"1c801be2-5b65-4113-9e33-764187508cc8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf" Sep 16 00:40:24.182165 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.182142 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc83d5a0-8676-49a2-ac3d-d992ded2b9da-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2\" (UID: \"cc83d5a0-8676-49a2-ac3d-d992ded2b9da\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2" Sep 16 00:40:24.182165 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.182165 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spz8r\" (UniqueName: \"kubernetes.io/projected/cc83d5a0-8676-49a2-ac3d-d992ded2b9da-kube-api-access-spz8r\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2\" (UID: \"cc83d5a0-8676-49a2-ac3d-d992ded2b9da\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2" Sep 16 00:40:24.182412 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.182183 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc83d5a0-8676-49a2-ac3d-d992ded2b9da-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2\" (UID: \"cc83d5a0-8676-49a2-ac3d-d992ded2b9da\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2" Sep 16 00:40:24.182412 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.182230 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7b9f\" (UniqueName: \"kubernetes.io/projected/1c801be2-5b65-4113-9e33-764187508cc8-kube-api-access-k7b9f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf\" (UID: \"1c801be2-5b65-4113-9e33-764187508cc8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf" Sep 16 00:40:24.182412 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.182267 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c801be2-5b65-4113-9e33-764187508cc8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf\" (UID: \"1c801be2-5b65-4113-9e33-764187508cc8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf" Sep 16 00:40:24.182564 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.182473 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c801be2-5b65-4113-9e33-764187508cc8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf\" (UID: \"1c801be2-5b65-4113-9e33-764187508cc8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf" Sep 16 00:40:24.182564 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.182552 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c801be2-5b65-4113-9e33-764187508cc8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf\" (UID: \"1c801be2-5b65-4113-9e33-764187508cc8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf" Sep 16 00:40:24.188243 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.188223 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd"] Sep 16 00:40:24.191564 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.191543 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7b9f\" (UniqueName: \"kubernetes.io/projected/1c801be2-5b65-4113-9e33-764187508cc8-kube-api-access-k7b9f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf\" (UID: \"1c801be2-5b65-4113-9e33-764187508cc8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf" Sep 16 00:40:24.191895 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.191878 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd" Sep 16 00:40:24.198926 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.198909 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd"] Sep 16 00:40:24.207085 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.207068 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm" Sep 16 00:40:24.283829 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.283795 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4d0065d-2721-4b07-8863-eb0a2af389b7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd\" (UID: \"b4d0065d-2721-4b07-8863-eb0a2af389b7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd" Sep 16 00:40:24.283970 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.283846 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gmjt\" (UniqueName: \"kubernetes.io/projected/b4d0065d-2721-4b07-8863-eb0a2af389b7-kube-api-access-8gmjt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd\" (UID: \"b4d0065d-2721-4b07-8863-eb0a2af389b7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd" Sep 16 00:40:24.283970 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.283919 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc83d5a0-8676-49a2-ac3d-d992ded2b9da-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2\" (UID: \"cc83d5a0-8676-49a2-ac3d-d992ded2b9da\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2" Sep 16 00:40:24.284171 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.283968 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spz8r\" (UniqueName: \"kubernetes.io/projected/cc83d5a0-8676-49a2-ac3d-d992ded2b9da-kube-api-access-spz8r\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2\" (UID: \"cc83d5a0-8676-49a2-ac3d-d992ded2b9da\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2" Sep 16 00:40:24.284171 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.283999 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc83d5a0-8676-49a2-ac3d-d992ded2b9da-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2\" (UID: \"cc83d5a0-8676-49a2-ac3d-d992ded2b9da\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2" Sep 16 00:40:24.284171 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.284027 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4d0065d-2721-4b07-8863-eb0a2af389b7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd\" (UID: \"b4d0065d-2721-4b07-8863-eb0a2af389b7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd" Sep 16 00:40:24.284379 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.284359 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc83d5a0-8676-49a2-ac3d-d992ded2b9da-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2\" (UID: \"cc83d5a0-8676-49a2-ac3d-d992ded2b9da\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2" Sep 16 00:40:24.284418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.284383 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc83d5a0-8676-49a2-ac3d-d992ded2b9da-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2\" (UID: \"cc83d5a0-8676-49a2-ac3d-d992ded2b9da\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2" Sep 16 00:40:24.292690 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.292663 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spz8r\" (UniqueName: \"kubernetes.io/projected/cc83d5a0-8676-49a2-ac3d-d992ded2b9da-kube-api-access-spz8r\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2\" (UID: \"cc83d5a0-8676-49a2-ac3d-d992ded2b9da\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2" Sep 16 00:40:24.305469 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.305450 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf" Sep 16 00:40:24.384710 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.384681 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4d0065d-2721-4b07-8863-eb0a2af389b7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd\" (UID: \"b4d0065d-2721-4b07-8863-eb0a2af389b7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd" Sep 16 00:40:24.384841 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.384720 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gmjt\" (UniqueName: \"kubernetes.io/projected/b4d0065d-2721-4b07-8863-eb0a2af389b7-kube-api-access-8gmjt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd\" (UID: \"b4d0065d-2721-4b07-8863-eb0a2af389b7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd" Sep 16 00:40:24.384962 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.384927 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4d0065d-2721-4b07-8863-eb0a2af389b7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd\" (UID: \"b4d0065d-2721-4b07-8863-eb0a2af389b7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd" Sep 16 00:40:24.385096 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.385075 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4d0065d-2721-4b07-8863-eb0a2af389b7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd\" (UID: \"b4d0065d-2721-4b07-8863-eb0a2af389b7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd" Sep 16 00:40:24.385266 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.385252 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4d0065d-2721-4b07-8863-eb0a2af389b7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd\" (UID: \"b4d0065d-2721-4b07-8863-eb0a2af389b7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd" Sep 16 00:40:24.393651 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.393630 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gmjt\" (UniqueName: \"kubernetes.io/projected/b4d0065d-2721-4b07-8863-eb0a2af389b7-kube-api-access-8gmjt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd\" (UID: \"b4d0065d-2721-4b07-8863-eb0a2af389b7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd" Sep 16 00:40:24.410583 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.410545 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2" Sep 16 00:40:24.483212 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.483180 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm"] Sep 16 00:40:24.483781 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:40:24.483752 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20df808_8e01_429a_8a27_cb9ee653c2e3.slice/crio-7bfff3eef77aec0f4a409ea88a679c3456c537af15db3843e462bc44360c7b5c WatchSource:0}: Error finding container 7bfff3eef77aec0f4a409ea88a679c3456c537af15db3843e462bc44360c7b5c: Status 404 returned error can't find the container with id 7bfff3eef77aec0f4a409ea88a679c3456c537af15db3843e462bc44360c7b5c Sep 16 00:40:24.506919 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.506899 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd" Sep 16 00:40:24.584786 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.584758 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf"] Sep 16 00:40:24.587934 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:40:24.587829 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c801be2_5b65_4113_9e33_764187508cc8.slice/crio-d53da7d5ccd455b8abf0ea7f88f47fc21a8aec6e3c5a83f2185be3c793689639 WatchSource:0}: Error finding container d53da7d5ccd455b8abf0ea7f88f47fc21a8aec6e3c5a83f2185be3c793689639: Status 404 returned error can't find the container with id d53da7d5ccd455b8abf0ea7f88f47fc21a8aec6e3c5a83f2185be3c793689639 Sep 16 00:40:24.682807 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.682784 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2"] Sep 16 00:40:24.685656 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:40:24.685627 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc83d5a0_8676_49a2_ac3d_d992ded2b9da.slice/crio-db2d64e295153754e0234bd9a2a52523cadfec90ba09bb9214efdde305e6b06f WatchSource:0}: Error finding container db2d64e295153754e0234bd9a2a52523cadfec90ba09bb9214efdde305e6b06f: Status 404 returned error can't find the container with id db2d64e295153754e0234bd9a2a52523cadfec90ba09bb9214efdde305e6b06f Sep 16 00:40:24.731097 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.731066 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2" event={"ID":"cc83d5a0-8676-49a2-ac3d-d992ded2b9da","Type":"ContainerStarted","Data":"db2d64e295153754e0234bd9a2a52523cadfec90ba09bb9214efdde305e6b06f"} Sep 16 00:40:24.732730 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.732706 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf" event={"ID":"1c801be2-5b65-4113-9e33-764187508cc8","Type":"ContainerStarted","Data":"ac675f6b22fa8be5f2569f7fe92896b6303d18a5bfebe243d9e1194af34cd3b2"} Sep 16 00:40:24.732832 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.732753 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf" event={"ID":"1c801be2-5b65-4113-9e33-764187508cc8","Type":"ContainerStarted","Data":"d53da7d5ccd455b8abf0ea7f88f47fc21a8aec6e3c5a83f2185be3c793689639"} Sep 16 00:40:24.734358 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.734305 2567 generic.go:356] "Generic (PLEG): container finished" podID="d20df808-8e01-429a-8a27-cb9ee653c2e3" containerID="4b37422248412c0d4ce060b20301e6ac0c753d4cb4709fd6ee1286c94abbe703" exitCode=0 Sep 16 00:40:24.734358 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.734338 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm" event={"ID":"d20df808-8e01-429a-8a27-cb9ee653c2e3","Type":"ContainerDied","Data":"4b37422248412c0d4ce060b20301e6ac0c753d4cb4709fd6ee1286c94abbe703"} Sep 16 00:40:24.734448 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.734367 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm" event={"ID":"d20df808-8e01-429a-8a27-cb9ee653c2e3","Type":"ContainerStarted","Data":"7bfff3eef77aec0f4a409ea88a679c3456c537af15db3843e462bc44360c7b5c"} Sep 16 00:40:24.779556 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:24.779534 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd"] Sep 16 00:40:24.843803 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:40:24.843775 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d0065d_2721_4b07_8863_eb0a2af389b7.slice/crio-24e5bacedf6a1d914847dbb2325ccac80d782af9138d3ca7c3b8a67d040f748f WatchSource:0}: Error finding container 24e5bacedf6a1d914847dbb2325ccac80d782af9138d3ca7c3b8a67d040f748f: Status 404 returned error can't find the container with id 24e5bacedf6a1d914847dbb2325ccac80d782af9138d3ca7c3b8a67d040f748f Sep 16 00:40:25.740593 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:25.740513 2567 generic.go:356] "Generic (PLEG): container finished" podID="cc83d5a0-8676-49a2-ac3d-d992ded2b9da" containerID="48c19f601c5fbab1a03765dec325a91296cccc55ea1f7f41c0696980fb3dcaba" exitCode=0 Sep 16 00:40:25.740593 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:25.740580 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2" event={"ID":"cc83d5a0-8676-49a2-ac3d-d992ded2b9da","Type":"ContainerDied","Data":"48c19f601c5fbab1a03765dec325a91296cccc55ea1f7f41c0696980fb3dcaba"} Sep 16 00:40:25.742202 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:25.742175 2567 generic.go:356] "Generic (PLEG): container finished" podID="b4d0065d-2721-4b07-8863-eb0a2af389b7" containerID="ea8e36db83528a44078e7d60fc57e3b9d3e2e438312737dc93316ec653198834" exitCode=0 Sep 16 00:40:25.742329 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:25.742210 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd" event={"ID":"b4d0065d-2721-4b07-8863-eb0a2af389b7","Type":"ContainerDied","Data":"ea8e36db83528a44078e7d60fc57e3b9d3e2e438312737dc93316ec653198834"} Sep 16 00:40:25.742329 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:25.742257 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd" event={"ID":"b4d0065d-2721-4b07-8863-eb0a2af389b7","Type":"ContainerStarted","Data":"24e5bacedf6a1d914847dbb2325ccac80d782af9138d3ca7c3b8a67d040f748f"} Sep 16 00:40:25.743801 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:25.743785 2567 generic.go:356] "Generic (PLEG): container finished" podID="1c801be2-5b65-4113-9e33-764187508cc8" containerID="ac675f6b22fa8be5f2569f7fe92896b6303d18a5bfebe243d9e1194af34cd3b2" exitCode=0 Sep 16 00:40:25.743870 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:25.743851 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf" event={"ID":"1c801be2-5b65-4113-9e33-764187508cc8","Type":"ContainerDied","Data":"ac675f6b22fa8be5f2569f7fe92896b6303d18a5bfebe243d9e1194af34cd3b2"} Sep 16 00:40:25.746015 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:25.745997 2567 generic.go:356] "Generic (PLEG): container finished" podID="d20df808-8e01-429a-8a27-cb9ee653c2e3" containerID="e352577417517477c85165a4aae974b6ddc4b33938a85e01056e6a19220cfcc2" exitCode=0 Sep 16 00:40:25.746132 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:25.746075 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm" event={"ID":"d20df808-8e01-429a-8a27-cb9ee653c2e3","Type":"ContainerDied","Data":"e352577417517477c85165a4aae974b6ddc4b33938a85e01056e6a19220cfcc2"} Sep 16 00:40:26.754502 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:26.754472 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2" event={"ID":"cc83d5a0-8676-49a2-ac3d-d992ded2b9da","Type":"ContainerStarted","Data":"61d186dbafc2baea37031cf5f178c10798d3ecdf28a4387206b749c37aeb2356"} Sep 16 00:40:26.756511 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:26.756485 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf" event={"ID":"1c801be2-5b65-4113-9e33-764187508cc8","Type":"ContainerStarted","Data":"71db6f1f1ded5ba69492245a49c1a8278ffec6a2766b79f95672b9d0f2f5aac8"} Sep 16 00:40:26.761259 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:26.758973 2567 generic.go:356] "Generic (PLEG): container finished" podID="d20df808-8e01-429a-8a27-cb9ee653c2e3" containerID="c318253e6b1d4804db90e0a67bac6ac4f843b6d55998d1f7a7348fc531d2527c" exitCode=0 Sep 16 00:40:26.761259 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:26.759020 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm" event={"ID":"d20df808-8e01-429a-8a27-cb9ee653c2e3","Type":"ContainerDied","Data":"c318253e6b1d4804db90e0a67bac6ac4f843b6d55998d1f7a7348fc531d2527c"} Sep 16 00:40:27.766921 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:27.766891 2567 generic.go:356] "Generic (PLEG): container finished" podID="cc83d5a0-8676-49a2-ac3d-d992ded2b9da" containerID="61d186dbafc2baea37031cf5f178c10798d3ecdf28a4387206b749c37aeb2356" exitCode=0 Sep 16 00:40:27.767345 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:27.766923 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2" event={"ID":"cc83d5a0-8676-49a2-ac3d-d992ded2b9da","Type":"ContainerDied","Data":"61d186dbafc2baea37031cf5f178c10798d3ecdf28a4387206b749c37aeb2356"} Sep 16 00:40:27.769151 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:27.769124 2567 generic.go:356] "Generic (PLEG): container finished" podID="b4d0065d-2721-4b07-8863-eb0a2af389b7" containerID="2e4bc5303a4a85120c358cb046b781f65660d5f66d89e5b287f48f0ec8b766d7" exitCode=0 Sep 16 00:40:27.769259 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:27.769215 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd" event={"ID":"b4d0065d-2721-4b07-8863-eb0a2af389b7","Type":"ContainerDied","Data":"2e4bc5303a4a85120c358cb046b781f65660d5f66d89e5b287f48f0ec8b766d7"} Sep 16 00:40:27.771480 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:27.771453 2567 generic.go:356] "Generic (PLEG): container finished" podID="1c801be2-5b65-4113-9e33-764187508cc8" containerID="71db6f1f1ded5ba69492245a49c1a8278ffec6a2766b79f95672b9d0f2f5aac8" exitCode=0 Sep 16 00:40:27.771593 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:27.771527 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf" event={"ID":"1c801be2-5b65-4113-9e33-764187508cc8","Type":"ContainerDied","Data":"71db6f1f1ded5ba69492245a49c1a8278ffec6a2766b79f95672b9d0f2f5aac8"} Sep 16 00:40:27.927825 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:27.927592 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm" Sep 16 00:40:28.119940 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:28.119918 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvrjc\" (UniqueName: \"kubernetes.io/projected/d20df808-8e01-429a-8a27-cb9ee653c2e3-kube-api-access-rvrjc\") pod \"d20df808-8e01-429a-8a27-cb9ee653c2e3\" (UID: \"d20df808-8e01-429a-8a27-cb9ee653c2e3\") " Sep 16 00:40:28.120094 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:28.119969 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d20df808-8e01-429a-8a27-cb9ee653c2e3-bundle\") pod \"d20df808-8e01-429a-8a27-cb9ee653c2e3\" (UID: \"d20df808-8e01-429a-8a27-cb9ee653c2e3\") " Sep 16 00:40:28.120094 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:28.119984 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d20df808-8e01-429a-8a27-cb9ee653c2e3-util\") pod \"d20df808-8e01-429a-8a27-cb9ee653c2e3\" (UID: \"d20df808-8e01-429a-8a27-cb9ee653c2e3\") " Sep 16 00:40:28.120382 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:28.120359 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20df808-8e01-429a-8a27-cb9ee653c2e3-bundle" (OuterVolumeSpecName: "bundle") pod "d20df808-8e01-429a-8a27-cb9ee653c2e3" (UID: "d20df808-8e01-429a-8a27-cb9ee653c2e3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:40:28.122016 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:28.121992 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20df808-8e01-429a-8a27-cb9ee653c2e3-kube-api-access-rvrjc" (OuterVolumeSpecName: "kube-api-access-rvrjc") pod "d20df808-8e01-429a-8a27-cb9ee653c2e3" (UID: "d20df808-8e01-429a-8a27-cb9ee653c2e3"). InnerVolumeSpecName "kube-api-access-rvrjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:40:28.125758 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:28.125727 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20df808-8e01-429a-8a27-cb9ee653c2e3-util" (OuterVolumeSpecName: "util") pod "d20df808-8e01-429a-8a27-cb9ee653c2e3" (UID: "d20df808-8e01-429a-8a27-cb9ee653c2e3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:40:28.220639 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:28.220617 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvrjc\" (UniqueName: \"kubernetes.io/projected/d20df808-8e01-429a-8a27-cb9ee653c2e3-kube-api-access-rvrjc\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:40:28.220639 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:28.220640 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d20df808-8e01-429a-8a27-cb9ee653c2e3-bundle\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:40:28.220752 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:28.220649 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d20df808-8e01-429a-8a27-cb9ee653c2e3-util\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:40:28.784406 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:28.784373 2567 generic.go:356] "Generic (PLEG): container finished" podID="cc83d5a0-8676-49a2-ac3d-d992ded2b9da" containerID="402a800ea60b79fd6a644d3e52ba8f6ac1984ea9d1cc5b023fa39f6259e4d39b" exitCode=0 Sep 16 00:40:28.784828 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:28.784468 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2" event={"ID":"cc83d5a0-8676-49a2-ac3d-d992ded2b9da","Type":"ContainerDied","Data":"402a800ea60b79fd6a644d3e52ba8f6ac1984ea9d1cc5b023fa39f6259e4d39b"} Sep 16 00:40:28.786601 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:28.786580 2567 generic.go:356] "Generic (PLEG): container finished" podID="b4d0065d-2721-4b07-8863-eb0a2af389b7" containerID="d5c12a61661b6e0faf6934b0c53ab6ee60d7323b92d8ddbe3fcb5bc74ceacc41" exitCode=0 Sep 16 00:40:28.786725 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:28.786647 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd" event={"ID":"b4d0065d-2721-4b07-8863-eb0a2af389b7","Type":"ContainerDied","Data":"d5c12a61661b6e0faf6934b0c53ab6ee60d7323b92d8ddbe3fcb5bc74ceacc41"} Sep 16 00:40:28.789072 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:28.789025 2567 generic.go:356] "Generic (PLEG): container finished" podID="1c801be2-5b65-4113-9e33-764187508cc8" containerID="c845dbef30a612098098f3fea7b597675cbfa290ffec79e84997d8452490a4b8" exitCode=0 Sep 16 00:40:28.789148 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:28.789068 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf" event={"ID":"1c801be2-5b65-4113-9e33-764187508cc8","Type":"ContainerDied","Data":"c845dbef30a612098098f3fea7b597675cbfa290ffec79e84997d8452490a4b8"} Sep 16 00:40:28.791127 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:28.791106 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm" event={"ID":"d20df808-8e01-429a-8a27-cb9ee653c2e3","Type":"ContainerDied","Data":"7bfff3eef77aec0f4a409ea88a679c3456c537af15db3843e462bc44360c7b5c"} Sep 16 00:40:28.791211 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:28.791144 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bfff3eef77aec0f4a409ea88a679c3456c537af15db3843e462bc44360c7b5c" Sep 16 00:40:28.791211 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:28.791149 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f65jm" Sep 16 00:40:29.948207 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:29.948183 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2" Sep 16 00:40:29.952483 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:29.952465 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf" Sep 16 00:40:29.979518 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:29.979496 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd" Sep 16 00:40:30.136260 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.136236 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spz8r\" (UniqueName: \"kubernetes.io/projected/cc83d5a0-8676-49a2-ac3d-d992ded2b9da-kube-api-access-spz8r\") pod \"cc83d5a0-8676-49a2-ac3d-d992ded2b9da\" (UID: \"cc83d5a0-8676-49a2-ac3d-d992ded2b9da\") " Sep 16 00:40:30.136418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.136274 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7b9f\" (UniqueName: \"kubernetes.io/projected/1c801be2-5b65-4113-9e33-764187508cc8-kube-api-access-k7b9f\") pod \"1c801be2-5b65-4113-9e33-764187508cc8\" (UID: \"1c801be2-5b65-4113-9e33-764187508cc8\") " Sep 16 00:40:30.136418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.136300 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c801be2-5b65-4113-9e33-764187508cc8-bundle\") pod \"1c801be2-5b65-4113-9e33-764187508cc8\" (UID: \"1c801be2-5b65-4113-9e33-764187508cc8\") " Sep 16 00:40:30.136418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.136340 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc83d5a0-8676-49a2-ac3d-d992ded2b9da-util\") pod \"cc83d5a0-8676-49a2-ac3d-d992ded2b9da\" (UID: \"cc83d5a0-8676-49a2-ac3d-d992ded2b9da\") " Sep 16 00:40:30.136418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.136367 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c801be2-5b65-4113-9e33-764187508cc8-util\") pod \"1c801be2-5b65-4113-9e33-764187508cc8\" (UID: \"1c801be2-5b65-4113-9e33-764187508cc8\") " Sep 16 00:40:30.136418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.136390 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4d0065d-2721-4b07-8863-eb0a2af389b7-bundle\") pod \"b4d0065d-2721-4b07-8863-eb0a2af389b7\" (UID: \"b4d0065d-2721-4b07-8863-eb0a2af389b7\") " Sep 16 00:40:30.136648 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.136448 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc83d5a0-8676-49a2-ac3d-d992ded2b9da-bundle\") pod \"cc83d5a0-8676-49a2-ac3d-d992ded2b9da\" (UID: \"cc83d5a0-8676-49a2-ac3d-d992ded2b9da\") " Sep 16 00:40:30.136648 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.136474 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gmjt\" (UniqueName: \"kubernetes.io/projected/b4d0065d-2721-4b07-8863-eb0a2af389b7-kube-api-access-8gmjt\") pod \"b4d0065d-2721-4b07-8863-eb0a2af389b7\" (UID: \"b4d0065d-2721-4b07-8863-eb0a2af389b7\") " Sep 16 00:40:30.137300 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.137197 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc83d5a0-8676-49a2-ac3d-d992ded2b9da-bundle" (OuterVolumeSpecName: "bundle") pod "cc83d5a0-8676-49a2-ac3d-d992ded2b9da" (UID: "cc83d5a0-8676-49a2-ac3d-d992ded2b9da"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:40:30.137300 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.137241 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c801be2-5b65-4113-9e33-764187508cc8-bundle" (OuterVolumeSpecName: "bundle") pod "1c801be2-5b65-4113-9e33-764187508cc8" (UID: "1c801be2-5b65-4113-9e33-764187508cc8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:40:30.137300 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.137268 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4d0065d-2721-4b07-8863-eb0a2af389b7-bundle" (OuterVolumeSpecName: "bundle") pod "b4d0065d-2721-4b07-8863-eb0a2af389b7" (UID: "b4d0065d-2721-4b07-8863-eb0a2af389b7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:40:30.137300 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.137295 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4d0065d-2721-4b07-8863-eb0a2af389b7-util\") pod \"b4d0065d-2721-4b07-8863-eb0a2af389b7\" (UID: \"b4d0065d-2721-4b07-8863-eb0a2af389b7\") " Sep 16 00:40:30.137919 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.137894 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c801be2-5b65-4113-9e33-764187508cc8-bundle\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:40:30.137919 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.137913 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4d0065d-2721-4b07-8863-eb0a2af389b7-bundle\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:40:30.138064 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.137926 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc83d5a0-8676-49a2-ac3d-d992ded2b9da-bundle\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:40:30.138896 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.138869 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc83d5a0-8676-49a2-ac3d-d992ded2b9da-kube-api-access-spz8r" (OuterVolumeSpecName: "kube-api-access-spz8r") pod "cc83d5a0-8676-49a2-ac3d-d992ded2b9da" (UID: "cc83d5a0-8676-49a2-ac3d-d992ded2b9da"). InnerVolumeSpecName "kube-api-access-spz8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:40:30.139055 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.139017 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c801be2-5b65-4113-9e33-764187508cc8-kube-api-access-k7b9f" (OuterVolumeSpecName: "kube-api-access-k7b9f") pod "1c801be2-5b65-4113-9e33-764187508cc8" (UID: "1c801be2-5b65-4113-9e33-764187508cc8"). InnerVolumeSpecName "kube-api-access-k7b9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:40:30.139987 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.139963 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d0065d-2721-4b07-8863-eb0a2af389b7-kube-api-access-8gmjt" (OuterVolumeSpecName: "kube-api-access-8gmjt") pod "b4d0065d-2721-4b07-8863-eb0a2af389b7" (UID: "b4d0065d-2721-4b07-8863-eb0a2af389b7"). InnerVolumeSpecName "kube-api-access-8gmjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:40:30.142996 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.142976 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc83d5a0-8676-49a2-ac3d-d992ded2b9da-util" (OuterVolumeSpecName: "util") pod "cc83d5a0-8676-49a2-ac3d-d992ded2b9da" (UID: "cc83d5a0-8676-49a2-ac3d-d992ded2b9da"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:40:30.143406 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.143376 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c801be2-5b65-4113-9e33-764187508cc8-util" (OuterVolumeSpecName: "util") pod "1c801be2-5b65-4113-9e33-764187508cc8" (UID: "1c801be2-5b65-4113-9e33-764187508cc8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:40:30.144685 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.144666 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4d0065d-2721-4b07-8863-eb0a2af389b7-util" (OuterVolumeSpecName: "util") pod "b4d0065d-2721-4b07-8863-eb0a2af389b7" (UID: "b4d0065d-2721-4b07-8863-eb0a2af389b7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:40:30.238892 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.238863 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spz8r\" (UniqueName: \"kubernetes.io/projected/cc83d5a0-8676-49a2-ac3d-d992ded2b9da-kube-api-access-spz8r\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:40:30.238892 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.238890 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7b9f\" (UniqueName: \"kubernetes.io/projected/1c801be2-5b65-4113-9e33-764187508cc8-kube-api-access-k7b9f\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:40:30.239077 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.238905 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc83d5a0-8676-49a2-ac3d-d992ded2b9da-util\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:40:30.239077 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.238916 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c801be2-5b65-4113-9e33-764187508cc8-util\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:40:30.239077 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.238927 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gmjt\" (UniqueName: \"kubernetes.io/projected/b4d0065d-2721-4b07-8863-eb0a2af389b7-kube-api-access-8gmjt\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:40:30.239077 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.238939 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4d0065d-2721-4b07-8863-eb0a2af389b7-util\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:40:30.802200 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.802168 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2" event={"ID":"cc83d5a0-8676-49a2-ac3d-d992ded2b9da","Type":"ContainerDied","Data":"db2d64e295153754e0234bd9a2a52523cadfec90ba09bb9214efdde305e6b06f"} Sep 16 00:40:30.802388 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.802225 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db2d64e295153754e0234bd9a2a52523cadfec90ba09bb9214efdde305e6b06f" Sep 16 00:40:30.802388 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.802181 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jrtz2" Sep 16 00:40:30.804230 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.804202 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd" Sep 16 00:40:30.804343 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.804201 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8cvfd" event={"ID":"b4d0065d-2721-4b07-8863-eb0a2af389b7","Type":"ContainerDied","Data":"24e5bacedf6a1d914847dbb2325ccac80d782af9138d3ca7c3b8a67d040f748f"} Sep 16 00:40:30.804401 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.804352 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24e5bacedf6a1d914847dbb2325ccac80d782af9138d3ca7c3b8a67d040f748f" Sep 16 00:40:30.806448 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.806429 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf" Sep 16 00:40:30.806566 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.806497 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfdjf" event={"ID":"1c801be2-5b65-4113-9e33-764187508cc8","Type":"ContainerDied","Data":"d53da7d5ccd455b8abf0ea7f88f47fc21a8aec6e3c5a83f2185be3c793689639"} Sep 16 00:40:30.806566 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:30.806533 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d53da7d5ccd455b8abf0ea7f88f47fc21a8aec6e3c5a83f2185be3c793689639" Sep 16 00:40:34.464589 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:34.464557 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:37.443620 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:37.443581 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:38.042780 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:38.042747 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:41.251587 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:41.251547 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:41.515961 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:41.515880 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:41.624801 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:41.624774 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:42.930421 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:42.930387 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:43.338617 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.338576 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-ff79b7c9f-qhnnv" podUID="94b239b5-f924-40c3-a828-85a7af826188" containerName="console" containerID="cri-o://c8205f2027d86863998297295a944fa53139c1faf8295ef57f6ece04d6349c91" gracePeriod=15 Sep 16 00:40:43.466450 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.466422 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:43.588790 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.588733 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:40:43.637625 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.637601 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-trusted-ca-bundle\") pod \"94b239b5-f924-40c3-a828-85a7af826188\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " Sep 16 00:40:43.637772 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.637631 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-console-config\") pod \"94b239b5-f924-40c3-a828-85a7af826188\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " Sep 16 00:40:43.637772 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.637665 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-service-ca\") pod \"94b239b5-f924-40c3-a828-85a7af826188\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " Sep 16 00:40:43.637772 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.637687 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94b239b5-f924-40c3-a828-85a7af826188-console-serving-cert\") pod \"94b239b5-f924-40c3-a828-85a7af826188\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " Sep 16 00:40:43.637772 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.637717 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94b239b5-f924-40c3-a828-85a7af826188-console-oauth-config\") pod \"94b239b5-f924-40c3-a828-85a7af826188\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " Sep 16 00:40:43.637772 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.637739 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-oauth-serving-cert\") pod \"94b239b5-f924-40c3-a828-85a7af826188\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " Sep 16 00:40:43.637772 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.637764 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bv9zv\" (UniqueName: \"kubernetes.io/projected/94b239b5-f924-40c3-a828-85a7af826188-kube-api-access-bv9zv\") pod \"94b239b5-f924-40c3-a828-85a7af826188\" (UID: \"94b239b5-f924-40c3-a828-85a7af826188\") " Sep 16 00:40:43.638083 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.638056 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "94b239b5-f924-40c3-a828-85a7af826188" (UID: "94b239b5-f924-40c3-a828-85a7af826188"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:40:43.638147 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.638085 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-service-ca" (OuterVolumeSpecName: "service-ca") pod "94b239b5-f924-40c3-a828-85a7af826188" (UID: "94b239b5-f924-40c3-a828-85a7af826188"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:40:43.638147 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.638117 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-console-config" (OuterVolumeSpecName: "console-config") pod "94b239b5-f924-40c3-a828-85a7af826188" (UID: "94b239b5-f924-40c3-a828-85a7af826188"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:40:43.638263 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.638178 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "94b239b5-f924-40c3-a828-85a7af826188" (UID: "94b239b5-f924-40c3-a828-85a7af826188"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:40:43.640076 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.640029 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b239b5-f924-40c3-a828-85a7af826188-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "94b239b5-f924-40c3-a828-85a7af826188" (UID: "94b239b5-f924-40c3-a828-85a7af826188"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 00:40:43.640168 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.640086 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b239b5-f924-40c3-a828-85a7af826188-kube-api-access-bv9zv" (OuterVolumeSpecName: "kube-api-access-bv9zv") pod "94b239b5-f924-40c3-a828-85a7af826188" (UID: "94b239b5-f924-40c3-a828-85a7af826188"). InnerVolumeSpecName "kube-api-access-bv9zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:40:43.640210 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.640184 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b239b5-f924-40c3-a828-85a7af826188-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "94b239b5-f924-40c3-a828-85a7af826188" (UID: "94b239b5-f924-40c3-a828-85a7af826188"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 00:40:43.738665 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.738640 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-service-ca\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:40:43.738665 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.738663 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94b239b5-f924-40c3-a828-85a7af826188-console-serving-cert\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:40:43.738795 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.738671 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94b239b5-f924-40c3-a828-85a7af826188-console-oauth-config\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:40:43.738795 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.738680 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-oauth-serving-cert\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:40:43.738795 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.738692 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv9zv\" (UniqueName: \"kubernetes.io/projected/94b239b5-f924-40c3-a828-85a7af826188-kube-api-access-bv9zv\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:40:43.738795 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.738700 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-trusted-ca-bundle\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:40:43.738795 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.738709 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94b239b5-f924-40c3-a828-85a7af826188-console-config\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:40:43.848301 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.848234 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:43.874717 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.874689 2567 generic.go:356] "Generic (PLEG): container finished" podID="94b239b5-f924-40c3-a828-85a7af826188" containerID="c8205f2027d86863998297295a944fa53139c1faf8295ef57f6ece04d6349c91" exitCode=0 Sep 16 00:40:43.874829 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.874741 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff79b7c9f-qhnnv" Sep 16 00:40:43.874829 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.874779 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff79b7c9f-qhnnv" event={"ID":"94b239b5-f924-40c3-a828-85a7af826188","Type":"ContainerDied","Data":"c8205f2027d86863998297295a944fa53139c1faf8295ef57f6ece04d6349c91"} Sep 16 00:40:43.874919 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.874832 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff79b7c9f-qhnnv" event={"ID":"94b239b5-f924-40c3-a828-85a7af826188","Type":"ContainerDied","Data":"0cb6c6452f2c3f1c688d1e431ffd46cc998132ebb9e8f516c113d0728ef08f5c"} Sep 16 00:40:43.874919 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.874859 2567 scope.go:117] "RemoveContainer" containerID="c8205f2027d86863998297295a944fa53139c1faf8295ef57f6ece04d6349c91" Sep 16 00:40:43.892116 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.892079 2567 scope.go:117] "RemoveContainer" containerID="c8205f2027d86863998297295a944fa53139c1faf8295ef57f6ece04d6349c91" Sep 16 00:40:43.892517 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:40:43.892495 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8205f2027d86863998297295a944fa53139c1faf8295ef57f6ece04d6349c91\": container with ID starting with c8205f2027d86863998297295a944fa53139c1faf8295ef57f6ece04d6349c91 not found: ID does not exist" containerID="c8205f2027d86863998297295a944fa53139c1faf8295ef57f6ece04d6349c91" Sep 16 00:40:43.892590 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.892524 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8205f2027d86863998297295a944fa53139c1faf8295ef57f6ece04d6349c91"} err="failed to get container status \"c8205f2027d86863998297295a944fa53139c1faf8295ef57f6ece04d6349c91\": rpc error: code = NotFound desc = could not find container \"c8205f2027d86863998297295a944fa53139c1faf8295ef57f6ece04d6349c91\": container with ID starting with c8205f2027d86863998297295a944fa53139c1faf8295ef57f6ece04d6349c91 not found: ID does not exist" Sep 16 00:40:43.902712 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.902691 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ff79b7c9f-qhnnv"] Sep 16 00:40:43.913581 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:43.913557 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-ff79b7c9f-qhnnv"] Sep 16 00:40:44.093775 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:44.093748 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:44.120730 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:44.120651 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94b239b5-f924-40c3-a828-85a7af826188" path="/var/lib/kubelet/pods/94b239b5-f924-40c3-a828-85a7af826188/volumes" Sep 16 00:40:44.236179 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:44.236152 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:45.099232 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:45.099203 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:48.856358 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:48.856322 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:48.969297 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:48.969268 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:53.490144 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:53.490113 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:53.591054 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:53.591009 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:55.104733 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:55.104702 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:55.210651 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:55.210614 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:56.499668 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:56.499641 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:40:56.597419 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:40:56.597393 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:01.225262 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:01.225232 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:01.324781 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:01.324746 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:07.873149 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:07.873112 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:07.977676 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:07.977646 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:13.248781 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:13.248748 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:13.349994 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:13.349957 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:13.659184 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:13.659149 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:13.715490 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:13.715466 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:17.063305 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:17.063276 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:17.186168 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:17.186137 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:20.825060 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:20.825003 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:20.927850 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:20.927820 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:21.025539 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:21.025512 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:35.003407 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:35.003317 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:35.103428 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:35.103398 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:48.615837 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.615795 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:48.717711 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.717674 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:48.820291 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.820241 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:48.944497 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944427 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:41:48.944740 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944729 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc83d5a0-8676-49a2-ac3d-d992ded2b9da" containerName="pull" Sep 16 00:41:48.944793 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944742 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc83d5a0-8676-49a2-ac3d-d992ded2b9da" containerName="pull" Sep 16 00:41:48.944793 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944755 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20df808-8e01-429a-8a27-cb9ee653c2e3" containerName="extract" Sep 16 00:41:48.944793 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944760 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20df808-8e01-429a-8a27-cb9ee653c2e3" containerName="extract" Sep 16 00:41:48.944793 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944767 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20df808-8e01-429a-8a27-cb9ee653c2e3" containerName="util" Sep 16 00:41:48.944793 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944772 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20df808-8e01-429a-8a27-cb9ee653c2e3" containerName="util" Sep 16 00:41:48.944793 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944778 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c801be2-5b65-4113-9e33-764187508cc8" containerName="util" Sep 16 00:41:48.944793 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944784 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c801be2-5b65-4113-9e33-764187508cc8" containerName="util" Sep 16 00:41:48.944793 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944794 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc83d5a0-8676-49a2-ac3d-d992ded2b9da" containerName="extract" Sep 16 00:41:48.945103 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944800 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc83d5a0-8676-49a2-ac3d-d992ded2b9da" containerName="extract" Sep 16 00:41:48.945103 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944807 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c801be2-5b65-4113-9e33-764187508cc8" containerName="pull" Sep 16 00:41:48.945103 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944812 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c801be2-5b65-4113-9e33-764187508cc8" containerName="pull" Sep 16 00:41:48.945103 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944816 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4d0065d-2721-4b07-8863-eb0a2af389b7" containerName="pull" Sep 16 00:41:48.945103 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944821 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4d0065d-2721-4b07-8863-eb0a2af389b7" containerName="pull" Sep 16 00:41:48.945103 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944827 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc83d5a0-8676-49a2-ac3d-d992ded2b9da" containerName="util" Sep 16 00:41:48.945103 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944832 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc83d5a0-8676-49a2-ac3d-d992ded2b9da" containerName="util" Sep 16 00:41:48.945103 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944837 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4d0065d-2721-4b07-8863-eb0a2af389b7" containerName="extract" Sep 16 00:41:48.945103 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944842 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4d0065d-2721-4b07-8863-eb0a2af389b7" containerName="extract" Sep 16 00:41:48.945103 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944852 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94b239b5-f924-40c3-a828-85a7af826188" containerName="console" Sep 16 00:41:48.945103 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944856 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="94b239b5-f924-40c3-a828-85a7af826188" containerName="console" Sep 16 00:41:48.945103 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944890 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4d0065d-2721-4b07-8863-eb0a2af389b7" containerName="util" Sep 16 00:41:48.945103 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944895 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4d0065d-2721-4b07-8863-eb0a2af389b7" containerName="util" Sep 16 00:41:48.945103 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944902 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20df808-8e01-429a-8a27-cb9ee653c2e3" containerName="pull" Sep 16 00:41:48.945103 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944906 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20df808-8e01-429a-8a27-cb9ee653c2e3" containerName="pull" Sep 16 00:41:48.945103 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944911 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c801be2-5b65-4113-9e33-764187508cc8" containerName="extract" Sep 16 00:41:48.945103 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.944975 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c801be2-5b65-4113-9e33-764187508cc8" containerName="extract" Sep 16 00:41:48.945778 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.945111 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc83d5a0-8676-49a2-ac3d-d992ded2b9da" containerName="extract" Sep 16 00:41:48.945778 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.945128 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20df808-8e01-429a-8a27-cb9ee653c2e3" containerName="extract" Sep 16 00:41:48.945778 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.945140 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4d0065d-2721-4b07-8863-eb0a2af389b7" containerName="extract" Sep 16 00:41:48.945778 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.945148 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c801be2-5b65-4113-9e33-764187508cc8" containerName="extract" Sep 16 00:41:48.945778 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.945156 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="94b239b5-f924-40c3-a828-85a7af826188" containerName="console" Sep 16 00:41:48.948263 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.948247 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-jlmxq" Sep 16 00:41:48.950964 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.950933 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 16 00:41:48.952124 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.952105 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 16 00:41:48.952213 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.952106 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 16 00:41:48.952213 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.952106 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6z2cb\"" Sep 16 00:41:48.956216 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.956182 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:41:48.982403 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:48.982380 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:41:49.126734 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:49.126708 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwrxh\" (UniqueName: \"kubernetes.io/projected/9f037143-52b9-4170-a3da-eccf4e9c2aa9-kube-api-access-mwrxh\") pod \"limitador-limitador-84994f7ffc-jlmxq\" (UID: \"9f037143-52b9-4170-a3da-eccf4e9c2aa9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jlmxq" Sep 16 00:41:49.126836 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:49.126754 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9f037143-52b9-4170-a3da-eccf4e9c2aa9-config-file\") pod \"limitador-limitador-84994f7ffc-jlmxq\" (UID: \"9f037143-52b9-4170-a3da-eccf4e9c2aa9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jlmxq" Sep 16 00:41:49.227559 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:49.227498 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9f037143-52b9-4170-a3da-eccf4e9c2aa9-config-file\") pod \"limitador-limitador-84994f7ffc-jlmxq\" (UID: \"9f037143-52b9-4170-a3da-eccf4e9c2aa9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jlmxq" Sep 16 00:41:49.227661 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:49.227559 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrxh\" (UniqueName: \"kubernetes.io/projected/9f037143-52b9-4170-a3da-eccf4e9c2aa9-kube-api-access-mwrxh\") pod \"limitador-limitador-84994f7ffc-jlmxq\" (UID: \"9f037143-52b9-4170-a3da-eccf4e9c2aa9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jlmxq" Sep 16 00:41:49.228090 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:49.228073 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9f037143-52b9-4170-a3da-eccf4e9c2aa9-config-file\") pod \"limitador-limitador-84994f7ffc-jlmxq\" (UID: \"9f037143-52b9-4170-a3da-eccf4e9c2aa9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jlmxq" Sep 16 00:41:49.235505 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:49.235483 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwrxh\" (UniqueName: \"kubernetes.io/projected/9f037143-52b9-4170-a3da-eccf4e9c2aa9-kube-api-access-mwrxh\") pod \"limitador-limitador-84994f7ffc-jlmxq\" (UID: \"9f037143-52b9-4170-a3da-eccf4e9c2aa9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jlmxq" Sep 16 00:41:49.259906 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:49.259884 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-jlmxq" Sep 16 00:41:49.423153 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:49.423119 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:49.520689 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:49.520612 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:49.533750 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:49.533727 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:41:49.535773 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:41:49.535749 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f037143_52b9_4170_a3da_eccf4e9c2aa9.slice/crio-f0613f0da05e6d0c9020c38ec7bcee7ed9431684d460b4297e7ab69b81e0e7f6 WatchSource:0}: Error finding container f0613f0da05e6d0c9020c38ec7bcee7ed9431684d460b4297e7ab69b81e0e7f6: Status 404 returned error can't find the container with id f0613f0da05e6d0c9020c38ec7bcee7ed9431684d460b4297e7ab69b81e0e7f6 Sep 16 00:41:50.193335 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:50.193297 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-jlmxq" event={"ID":"9f037143-52b9-4170-a3da-eccf4e9c2aa9","Type":"ContainerStarted","Data":"f0613f0da05e6d0c9020c38ec7bcee7ed9431684d460b4297e7ab69b81e0e7f6"} Sep 16 00:41:52.207798 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:52.207707 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-jlmxq" event={"ID":"9f037143-52b9-4170-a3da-eccf4e9c2aa9","Type":"ContainerStarted","Data":"12375cbee797154a26a7d5d244462a1330092009074167961645e71e65ff006d"} Sep 16 00:41:52.208175 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:52.207823 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-jlmxq" Sep 16 00:41:52.224374 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:52.224325 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-jlmxq" podStartSLOduration=1.820854337 podStartE2EDuration="4.224305797s" totalImagesPullingTime="2.40345146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:41:52.223246715 +0000 UTC m=+462.585470268" watchObservedRunningTime="2026-09-16 00:41:52.224305797 +0000 UTC m=+462.586529340" Sep 16 00:41:53.601394 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:53.601360 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:41:53.705627 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:41:53.705600 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:42:03.213605 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:42:03.213577 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-jlmxq" Sep 16 00:42:06.216188 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:42:06.216158 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:42:06.315710 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:42:06.315678 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:42:06.708196 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:42:06.708151 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:42:19.256084 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:42:19.256052 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:42:19.504943 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:42:19.504914 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:42:26.635795 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:42:26.635763 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:42:26.736475 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:42:26.736437 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:42:26.788446 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:42:26.788423 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:42:26.915907 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:42:26.915830 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:42:27.156090 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:42:27.156059 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:42:30.610272 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:42:30.610235 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:42:30.708574 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:42:30.708545 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:42:45.273083 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:42:45.273006 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:43:02.141739 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:43:02.141632 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:43:02.250380 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:43:02.250353 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:43:03.617878 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:43:03.617829 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:43:03.768545 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:43:03.768515 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:43:03.937887 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:43:03.937809 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:43:04.041392 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:43:04.041349 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:43:04.258557 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:43:04.258485 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:43:04.357756 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:43:04.357722 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:43:04.507128 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:43:04.507094 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:43:16.263094 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:43:16.263021 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:43:16.364024 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:43:16.363990 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:43:34.713243 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:43:34.713181 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:43:35.053166 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:43:35.053134 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:43:38.818543 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:43:38.818503 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:43:50.316664 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:43:50.316631 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:43:56.092853 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:43:56.092801 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:43:56.743080 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:43:56.743030 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:43:56.758015 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:43:56.757986 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:43:57.787689 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:43:57.787654 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:44:24.883317 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:24.883286 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d"] Sep 16 00:44:24.886841 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:24.886821 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:24.889979 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:24.889959 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 16 00:44:24.890986 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:24.890966 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lbjkk\"" Sep 16 00:44:24.890986 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:24.890976 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 16 00:44:24.891179 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:24.891015 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 16 00:44:24.897326 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:24.897305 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d"] Sep 16 00:44:24.911874 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:24.911845 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:44:24.982329 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:24.982302 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97788523-6ee8-42cc-99ca-e9b65ed113a5-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d\" (UID: \"97788523-6ee8-42cc-99ca-e9b65ed113a5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:24.982438 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:24.982334 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfswh\" (UniqueName: \"kubernetes.io/projected/97788523-6ee8-42cc-99ca-e9b65ed113a5-kube-api-access-kfswh\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d\" (UID: \"97788523-6ee8-42cc-99ca-e9b65ed113a5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:24.982438 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:24.982377 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97788523-6ee8-42cc-99ca-e9b65ed113a5-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d\" (UID: \"97788523-6ee8-42cc-99ca-e9b65ed113a5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:24.982438 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:24.982422 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97788523-6ee8-42cc-99ca-e9b65ed113a5-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d\" (UID: \"97788523-6ee8-42cc-99ca-e9b65ed113a5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:24.982586 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:24.982464 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97788523-6ee8-42cc-99ca-e9b65ed113a5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d\" (UID: \"97788523-6ee8-42cc-99ca-e9b65ed113a5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:24.982586 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:24.982505 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97788523-6ee8-42cc-99ca-e9b65ed113a5-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d\" (UID: \"97788523-6ee8-42cc-99ca-e9b65ed113a5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:25.083796 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:25.083766 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97788523-6ee8-42cc-99ca-e9b65ed113a5-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d\" (UID: \"97788523-6ee8-42cc-99ca-e9b65ed113a5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:25.083940 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:25.083799 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfswh\" (UniqueName: \"kubernetes.io/projected/97788523-6ee8-42cc-99ca-e9b65ed113a5-kube-api-access-kfswh\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d\" (UID: \"97788523-6ee8-42cc-99ca-e9b65ed113a5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:25.083940 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:25.083826 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97788523-6ee8-42cc-99ca-e9b65ed113a5-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d\" (UID: \"97788523-6ee8-42cc-99ca-e9b65ed113a5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:25.083940 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:25.083851 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97788523-6ee8-42cc-99ca-e9b65ed113a5-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d\" (UID: \"97788523-6ee8-42cc-99ca-e9b65ed113a5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:25.083940 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:25.083889 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97788523-6ee8-42cc-99ca-e9b65ed113a5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d\" (UID: \"97788523-6ee8-42cc-99ca-e9b65ed113a5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:25.083940 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:25.083922 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97788523-6ee8-42cc-99ca-e9b65ed113a5-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d\" (UID: \"97788523-6ee8-42cc-99ca-e9b65ed113a5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:25.084322 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:25.084301 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97788523-6ee8-42cc-99ca-e9b65ed113a5-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d\" (UID: \"97788523-6ee8-42cc-99ca-e9b65ed113a5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:25.084378 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:25.084354 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97788523-6ee8-42cc-99ca-e9b65ed113a5-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d\" (UID: \"97788523-6ee8-42cc-99ca-e9b65ed113a5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:25.084378 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:25.084365 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97788523-6ee8-42cc-99ca-e9b65ed113a5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d\" (UID: \"97788523-6ee8-42cc-99ca-e9b65ed113a5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:25.086260 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:25.086239 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97788523-6ee8-42cc-99ca-e9b65ed113a5-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d\" (UID: \"97788523-6ee8-42cc-99ca-e9b65ed113a5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:25.086504 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:25.086487 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97788523-6ee8-42cc-99ca-e9b65ed113a5-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d\" (UID: \"97788523-6ee8-42cc-99ca-e9b65ed113a5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:25.091717 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:25.091698 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfswh\" (UniqueName: \"kubernetes.io/projected/97788523-6ee8-42cc-99ca-e9b65ed113a5-kube-api-access-kfswh\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d\" (UID: \"97788523-6ee8-42cc-99ca-e9b65ed113a5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:25.198131 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:25.198056 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:25.469128 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:25.469104 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d"] Sep 16 00:44:25.470462 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:44:25.470439 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97788523_6ee8_42cc_99ca_e9b65ed113a5.slice/crio-cf8a8964ab0a7b82ce02852c15c46e79028d423d4ef2477d02b175781624d7f0 WatchSource:0}: Error finding container cf8a8964ab0a7b82ce02852c15c46e79028d423d4ef2477d02b175781624d7f0: Status 404 returned error can't find the container with id cf8a8964ab0a7b82ce02852c15c46e79028d423d4ef2477d02b175781624d7f0 Sep 16 00:44:25.471963 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:25.471942 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 00:44:25.740433 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:25.740358 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:44:25.986343 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:25.986308 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" event={"ID":"97788523-6ee8-42cc-99ca-e9b65ed113a5","Type":"ContainerStarted","Data":"cf8a8964ab0a7b82ce02852c15c46e79028d423d4ef2477d02b175781624d7f0"} Sep 16 00:44:26.992344 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:26.992311 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" event={"ID":"97788523-6ee8-42cc-99ca-e9b65ed113a5","Type":"ContainerStarted","Data":"6acf5fb2a17fbae7cb678427a8d92cf4e1ce8a0541fd199d1c227555bef9ba6a"} Sep 16 00:44:27.009582 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:27.009538 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" podStartSLOduration=2.037882522 podStartE2EDuration="3.009521448s" totalImagesPullingTime="971.638926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:44:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:44:27.008204418 +0000 UTC m=+617.370427976" watchObservedRunningTime="2026-09-16 00:44:27.009521448 +0000 UTC m=+617.371745051" Sep 16 00:44:33.736455 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:33.736385 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:44:34.940207 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:34.940170 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:44:35.198703 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:35.198628 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:35.198703 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:35.198687 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:35.210804 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:35.210784 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:36.046740 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:36.046713 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dvm6d" Sep 16 00:44:44.991905 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:44.991875 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:44:46.536484 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:46.536452 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:44:59.885964 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:59.885923 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf"] Sep 16 00:44:59.889617 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:59.889596 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:44:59.892143 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:59.892118 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 16 00:44:59.897323 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:59.897305 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf"] Sep 16 00:44:59.909438 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:59.909418 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:44:59.952136 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:59.952112 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2070b7-4da8-48e8-85d0-5e11d45da928-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkfpf\" (UID: \"aa2070b7-4da8-48e8-85d0-5e11d45da928\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:44:59.952226 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:59.952154 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2070b7-4da8-48e8-85d0-5e11d45da928-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkfpf\" (UID: \"aa2070b7-4da8-48e8-85d0-5e11d45da928\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:44:59.952226 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:59.952180 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa2070b7-4da8-48e8-85d0-5e11d45da928-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkfpf\" (UID: \"aa2070b7-4da8-48e8-85d0-5e11d45da928\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:44:59.952226 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:59.952205 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2070b7-4da8-48e8-85d0-5e11d45da928-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkfpf\" (UID: \"aa2070b7-4da8-48e8-85d0-5e11d45da928\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:44:59.952226 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:59.952220 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2070b7-4da8-48e8-85d0-5e11d45da928-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkfpf\" (UID: \"aa2070b7-4da8-48e8-85d0-5e11d45da928\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:44:59.952403 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:44:59.952236 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hw4f\" (UniqueName: \"kubernetes.io/projected/aa2070b7-4da8-48e8-85d0-5e11d45da928-kube-api-access-9hw4f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkfpf\" (UID: \"aa2070b7-4da8-48e8-85d0-5e11d45da928\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:45:00.053480 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:00.053453 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2070b7-4da8-48e8-85d0-5e11d45da928-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkfpf\" (UID: \"aa2070b7-4da8-48e8-85d0-5e11d45da928\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:45:00.053596 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:00.053497 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2070b7-4da8-48e8-85d0-5e11d45da928-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkfpf\" (UID: \"aa2070b7-4da8-48e8-85d0-5e11d45da928\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:45:00.053596 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:00.053516 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa2070b7-4da8-48e8-85d0-5e11d45da928-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkfpf\" (UID: \"aa2070b7-4da8-48e8-85d0-5e11d45da928\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:45:00.053596 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:00.053540 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2070b7-4da8-48e8-85d0-5e11d45da928-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkfpf\" (UID: \"aa2070b7-4da8-48e8-85d0-5e11d45da928\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:45:00.053596 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:00.053565 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2070b7-4da8-48e8-85d0-5e11d45da928-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkfpf\" (UID: \"aa2070b7-4da8-48e8-85d0-5e11d45da928\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:45:00.053596 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:00.053591 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hw4f\" (UniqueName: \"kubernetes.io/projected/aa2070b7-4da8-48e8-85d0-5e11d45da928-kube-api-access-9hw4f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkfpf\" (UID: \"aa2070b7-4da8-48e8-85d0-5e11d45da928\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:45:00.053871 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:00.053848 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2070b7-4da8-48e8-85d0-5e11d45da928-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkfpf\" (UID: \"aa2070b7-4da8-48e8-85d0-5e11d45da928\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:45:00.053972 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:00.053938 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa2070b7-4da8-48e8-85d0-5e11d45da928-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkfpf\" (UID: \"aa2070b7-4da8-48e8-85d0-5e11d45da928\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:45:00.054102 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:00.053990 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2070b7-4da8-48e8-85d0-5e11d45da928-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkfpf\" (UID: \"aa2070b7-4da8-48e8-85d0-5e11d45da928\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:45:00.055921 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:00.055895 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2070b7-4da8-48e8-85d0-5e11d45da928-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkfpf\" (UID: \"aa2070b7-4da8-48e8-85d0-5e11d45da928\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:45:00.056301 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:00.056283 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2070b7-4da8-48e8-85d0-5e11d45da928-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkfpf\" (UID: \"aa2070b7-4da8-48e8-85d0-5e11d45da928\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:45:00.060962 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:00.060941 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hw4f\" (UniqueName: \"kubernetes.io/projected/aa2070b7-4da8-48e8-85d0-5e11d45da928-kube-api-access-9hw4f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkfpf\" (UID: \"aa2070b7-4da8-48e8-85d0-5e11d45da928\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:45:00.132665 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:00.132639 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825325-drbll"] Sep 16 00:45:00.136145 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:00.136105 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825325-drbll" Sep 16 00:45:00.138689 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:00.138664 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-npltg\"" Sep 16 00:45:00.142273 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:00.142250 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825325-drbll"] Sep 16 00:45:00.147405 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:00.147388 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825325-drbll" Sep 16 00:45:00.200373 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:00.200264 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:45:00.423785 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:00.423758 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825325-drbll"] Sep 16 00:45:00.425692 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:45:00.425656 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8c408c_11ef_4674_ae6f_c4c88dc3a2f8.slice/crio-0c6f75e33d1fbd7581891810ba6dec137fa595246efddd6f20c97bcf1a508c99 WatchSource:0}: Error finding container 0c6f75e33d1fbd7581891810ba6dec137fa595246efddd6f20c97bcf1a508c99: Status 404 returned error can't find the container with id 0c6f75e33d1fbd7581891810ba6dec137fa595246efddd6f20c97bcf1a508c99 Sep 16 00:45:00.480943 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:00.480921 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf"] Sep 16 00:45:00.482531 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:45:00.482495 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2070b7_4da8_48e8_85d0_5e11d45da928.slice/crio-7513f1579d394601bf1dd6600da04f629d707da9c70e1cfc79210fbc0d127e98 WatchSource:0}: Error finding container 7513f1579d394601bf1dd6600da04f629d707da9c70e1cfc79210fbc0d127e98: Status 404 returned error can't find the container with id 7513f1579d394601bf1dd6600da04f629d707da9c70e1cfc79210fbc0d127e98 Sep 16 00:45:01.161450 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:01.161420 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825325-drbll" event={"ID":"ba8c408c-11ef-4674-ae6f-c4c88dc3a2f8","Type":"ContainerStarted","Data":"0c6f75e33d1fbd7581891810ba6dec137fa595246efddd6f20c97bcf1a508c99"} Sep 16 00:45:01.162791 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:01.162769 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" event={"ID":"aa2070b7-4da8-48e8-85d0-5e11d45da928","Type":"ContainerStarted","Data":"d6143691c47960833d43581584d106f3fe63a605b856482f5cf97d5787b0dc38"} Sep 16 00:45:01.162876 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:01.162808 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" event={"ID":"aa2070b7-4da8-48e8-85d0-5e11d45da928","Type":"ContainerStarted","Data":"7513f1579d394601bf1dd6600da04f629d707da9c70e1cfc79210fbc0d127e98"} Sep 16 00:45:01.183433 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:01.183390 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" podStartSLOduration=1.988894894 podStartE2EDuration="2.183376224s" totalImagesPullingTime="194.48133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:45:01.18140558 +0000 UTC m=+651.543629112" watchObservedRunningTime="2026-09-16 00:45:01.183376224 +0000 UTC m=+651.545599756" Sep 16 00:45:01.640861 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:01.640830 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:45:04.178958 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:04.178867 2567 generic.go:356] "Generic (PLEG): container finished" podID="ba8c408c-11ef-4674-ae6f-c4c88dc3a2f8" containerID="0f9f422f698ed9dd498db600549c6834150567e12f4087b4b5cda52410352430" exitCode=0 Sep 16 00:45:04.178958 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:04.178924 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825325-drbll" event={"ID":"ba8c408c-11ef-4674-ae6f-c4c88dc3a2f8","Type":"ContainerDied","Data":"0f9f422f698ed9dd498db600549c6834150567e12f4087b4b5cda52410352430"} Sep 16 00:45:05.309528 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:05.309505 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825325-drbll" Sep 16 00:45:06.190000 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:06.189978 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825325-drbll" Sep 16 00:45:06.190181 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:06.189969 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825325-drbll" event={"ID":"ba8c408c-11ef-4674-ae6f-c4c88dc3a2f8","Type":"ContainerDied","Data":"0c6f75e33d1fbd7581891810ba6dec137fa595246efddd6f20c97bcf1a508c99"} Sep 16 00:45:06.190181 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:06.190107 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c6f75e33d1fbd7581891810ba6dec137fa595246efddd6f20c97bcf1a508c99" Sep 16 00:45:10.200471 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:10.200433 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:45:10.200868 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:10.200498 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:45:10.214866 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:10.214840 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:45:10.226175 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:10.226158 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkfpf" Sep 16 00:45:12.891149 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:12.891108 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:45:12.943001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:12.942978 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:45:20.621372 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:20.621342 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:45:22.842297 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:22.842249 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:45:24.640176 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:45:24.640140 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:46:33.503526 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:46:33.503451 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:46:33.988544 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:46:33.988514 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:46:34.116059 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:46:34.116006 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:46:50.289476 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:46:50.289446 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:46:50.383502 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:46:50.383470 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:15.761780 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:15.761744 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:16.116637 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:16.116583 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:17.081681 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:17.081649 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:17.182228 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:17.182195 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:18.260186 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:18.260152 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:18.358472 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:18.358443 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:34.655475 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:34.655365 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:34.756258 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:34.756216 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:34.992935 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:34.992837 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:35.092555 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.092518 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:35.097209 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.097179 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn"] Sep 16 00:47:35.097771 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.097744 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba8c408c-11ef-4674-ae6f-c4c88dc3a2f8" containerName="cleanup" Sep 16 00:47:35.097771 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.097772 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba8c408c-11ef-4674-ae6f-c4c88dc3a2f8" containerName="cleanup" Sep 16 00:47:35.097899 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.097861 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba8c408c-11ef-4674-ae6f-c4c88dc3a2f8" containerName="cleanup" Sep 16 00:47:35.101180 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.101159 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.104170 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.104142 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-5270e2-openshift-default-dockercfg-2vmhk\"" Sep 16 00:47:35.111869 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.111845 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn"] Sep 16 00:47:35.161099 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.161024 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:35.200332 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.200303 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-workload-certs\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.200467 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.200335 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd7jn\" (UniqueName: \"kubernetes.io/projected/3eff9577-aa6f-4e10-8538-ba0d09f132eb-kube-api-access-sd7jn\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.200467 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.200422 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-envoy\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.200467 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.200452 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-workload-socket\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.200587 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.200481 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-credential-socket\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.200587 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.200499 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-podinfo\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.200587 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.200520 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-token\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.200587 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.200557 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istiod-ca-cert\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.200729 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.200626 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-data\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.301470 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.301442 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-data\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.301624 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.301484 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-workload-certs\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.301624 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.301505 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd7jn\" (UniqueName: \"kubernetes.io/projected/3eff9577-aa6f-4e10-8538-ba0d09f132eb-kube-api-access-sd7jn\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.301624 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.301557 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-envoy\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.301624 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.301574 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-workload-socket\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.301624 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.301603 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-credential-socket\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.301873 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.301629 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-podinfo\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.301873 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.301651 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-token\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.301873 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.301677 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istiod-ca-cert\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.302002 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.301875 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-workload-certs\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.302002 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.301941 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-data\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.302002 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.301968 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-workload-socket\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.302193 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.302032 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-credential-socket\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.302347 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.302329 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istiod-ca-cert\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.304293 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.304258 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-envoy\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.304412 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.304396 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-podinfo\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.315921 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.315895 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-token\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.316209 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.316072 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd7jn\" (UniqueName: \"kubernetes.io/projected/3eff9577-aa6f-4e10-8538-ba0d09f132eb-kube-api-access-sd7jn\") pod \"e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.412420 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.412394 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:35.478672 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.478642 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:35.618379 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.618300 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:35.693773 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.693724 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn"] Sep 16 00:47:35.695783 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:47:35.695747 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eff9577_aa6f_4e10_8538_ba0d09f132eb.slice/crio-bd4acee0519959d7e632847db9ba9451c5c2d917623ae63e2b07e35bd12b25ef WatchSource:0}: Error finding container bd4acee0519959d7e632847db9ba9451c5c2d917623ae63e2b07e35bd12b25ef: Status 404 returned error can't find the container with id bd4acee0519959d7e632847db9ba9451c5c2d917623ae63e2b07e35bd12b25ef Sep 16 00:47:35.697900 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.697871 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:47:35.697980 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.697933 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:47:35.697980 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.697960 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:47:35.735723 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.735701 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:35.937724 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.937583 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" event={"ID":"3eff9577-aa6f-4e10-8538-ba0d09f132eb","Type":"ContainerStarted","Data":"ff74aa81986dfeb5bb051c9a0cf24d8dc0f4a116526a28b9e1d4481998ca664c"} Sep 16 00:47:35.937724 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.937637 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" event={"ID":"3eff9577-aa6f-4e10-8538-ba0d09f132eb","Type":"ContainerStarted","Data":"bd4acee0519959d7e632847db9ba9451c5c2d917623ae63e2b07e35bd12b25ef"} Sep 16 00:47:35.957443 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:35.957388 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" podStartSLOduration=0.957373899 podStartE2EDuration="957.373899ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:35.955508449 +0000 UTC m=+806.317732010" watchObservedRunningTime="2026-09-16 00:47:35.957373899 +0000 UTC m=+806.319597432" Sep 16 00:47:36.413210 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:36.413175 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:36.418502 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:36.418478 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:36.943830 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:36.943800 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:36.944728 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:36.944707 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:37.667183 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:37.667140 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:37.757684 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:37.757655 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:38.027996 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:38.027908 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:38.125163 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:38.125131 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:38.652414 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:38.652379 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:38.747126 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:38.747095 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:38.848463 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:38.848431 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:39.107055 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:39.107011 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:39.208324 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:39.208290 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:39.808863 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:39.808812 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:39.909774 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:39.909740 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:39.975950 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:39.975919 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:40.012325 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:40.012300 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg"] Sep 16 00:47:40.015728 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:40.015705 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg" Sep 16 00:47:40.018758 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:40.018728 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-66ecb7\"" Sep 16 00:47:40.018962 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:40.018948 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-66ecb7-dockercfg-bdqhf\"" Sep 16 00:47:40.028152 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:40.028132 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg"] Sep 16 00:47:40.144383 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:40.144301 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vst4\" (UniqueName: \"kubernetes.io/projected/fe6fcd3b-33cf-4706-af7f-54b182eb6821-kube-api-access-6vst4\") pod \"payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg\" (UID: \"fe6fcd3b-33cf-4706-af7f-54b182eb6821\") " pod="openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg" Sep 16 00:47:40.144383 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:40.144339 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe6fcd3b-33cf-4706-af7f-54b182eb6821-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg\" (UID: \"fe6fcd3b-33cf-4706-af7f-54b182eb6821\") " pod="openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg" Sep 16 00:47:40.244958 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:40.244923 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vst4\" (UniqueName: \"kubernetes.io/projected/fe6fcd3b-33cf-4706-af7f-54b182eb6821-kube-api-access-6vst4\") pod \"payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg\" (UID: \"fe6fcd3b-33cf-4706-af7f-54b182eb6821\") " pod="openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg" Sep 16 00:47:40.245187 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:40.244973 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe6fcd3b-33cf-4706-af7f-54b182eb6821-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg\" (UID: \"fe6fcd3b-33cf-4706-af7f-54b182eb6821\") " pod="openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg" Sep 16 00:47:40.245915 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:40.245889 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe6fcd3b-33cf-4706-af7f-54b182eb6821-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg\" (UID: \"fe6fcd3b-33cf-4706-af7f-54b182eb6821\") " pod="openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg" Sep 16 00:47:40.253545 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:40.253524 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vst4\" (UniqueName: \"kubernetes.io/projected/fe6fcd3b-33cf-4706-af7f-54b182eb6821-kube-api-access-6vst4\") pod \"payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg\" (UID: \"fe6fcd3b-33cf-4706-af7f-54b182eb6821\") " pod="openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg" Sep 16 00:47:40.326594 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:40.326568 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg" Sep 16 00:47:40.545574 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:40.545503 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:40.603082 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:40.603059 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg"] Sep 16 00:47:40.604649 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:47:40.604611 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6fcd3b_33cf_4706_af7f_54b182eb6821.slice/crio-1c04cd490acfc89520792bdee65429131720c8d617a9f6a0b139caf491d5e448 WatchSource:0}: Error finding container 1c04cd490acfc89520792bdee65429131720c8d617a9f6a0b139caf491d5e448: Status 404 returned error can't find the container with id 1c04cd490acfc89520792bdee65429131720c8d617a9f6a0b139caf491d5e448 Sep 16 00:47:40.604902 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:40.604881 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:40.968882 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:40.968838 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg" event={"ID":"fe6fcd3b-33cf-4706-af7f-54b182eb6821","Type":"ContainerStarted","Data":"1c04cd490acfc89520792bdee65429131720c8d617a9f6a0b139caf491d5e448"} Sep 16 00:47:41.008878 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:41.008824 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:41.110739 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:41.110703 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:41.436089 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:41.436060 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:41.487404 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:41.487376 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:41.910351 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:41.910324 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:41.979753 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:41.979720 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg" event={"ID":"fe6fcd3b-33cf-4706-af7f-54b182eb6821","Type":"ContainerStarted","Data":"47d2f75cc1739b53d2d31551b21dc17396ed91f072946d686a6eb91700eee6f1"} Sep 16 00:47:41.979914 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:41.979818 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg" Sep 16 00:47:41.998603 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:41.998553 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg" podStartSLOduration=1.780879466 podStartE2EDuration="2.998539553s" totalImagesPullingTime="1.217660087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:41.997004259 +0000 UTC m=+812.359227812" watchObservedRunningTime="2026-09-16 00:47:41.998539553 +0000 UTC m=+812.360763076" Sep 16 00:47:42.004647 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:42.004624 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x"] Sep 16 00:47:42.006621 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:42.006596 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:42.008231 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:42.008217 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x" Sep 16 00:47:42.010892 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:42.010868 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-0f7c3b\"" Sep 16 00:47:42.019145 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:42.019122 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x"] Sep 16 00:47:42.164615 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:42.164537 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2b07f9-744c-43d7-a97a-a6585938e0ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x\" (UID: \"8e2b07f9-744c-43d7-a97a-a6585938e0ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x" Sep 16 00:47:42.164615 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:42.164606 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhdpm\" (UniqueName: \"kubernetes.io/projected/8e2b07f9-744c-43d7-a97a-a6585938e0ee-kube-api-access-vhdpm\") pod \"payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x\" (UID: \"8e2b07f9-744c-43d7-a97a-a6585938e0ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x" Sep 16 00:47:42.265130 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:42.265100 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhdpm\" (UniqueName: \"kubernetes.io/projected/8e2b07f9-744c-43d7-a97a-a6585938e0ee-kube-api-access-vhdpm\") pod \"payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x\" (UID: \"8e2b07f9-744c-43d7-a97a-a6585938e0ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x" Sep 16 00:47:42.265305 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:42.265173 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2b07f9-744c-43d7-a97a-a6585938e0ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x\" (UID: \"8e2b07f9-744c-43d7-a97a-a6585938e0ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x" Sep 16 00:47:42.265881 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:42.265863 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2b07f9-744c-43d7-a97a-a6585938e0ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x\" (UID: \"8e2b07f9-744c-43d7-a97a-a6585938e0ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x" Sep 16 00:47:42.272782 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:42.272759 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhdpm\" (UniqueName: \"kubernetes.io/projected/8e2b07f9-744c-43d7-a97a-a6585938e0ee-kube-api-access-vhdpm\") pod \"payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x\" (UID: \"8e2b07f9-744c-43d7-a97a-a6585938e0ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x" Sep 16 00:47:42.320626 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:42.320599 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x" Sep 16 00:47:42.434554 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:42.434487 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:42.594733 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:42.594704 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x"] Sep 16 00:47:42.595817 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:47:42.595793 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2b07f9_744c_43d7_a97a_a6585938e0ee.slice/crio-6b2719a78e8f56ebafbe81f673fe6dd5f56e931104610e6027a8ecabc0661527 WatchSource:0}: Error finding container 6b2719a78e8f56ebafbe81f673fe6dd5f56e931104610e6027a8ecabc0661527: Status 404 returned error can't find the container with id 6b2719a78e8f56ebafbe81f673fe6dd5f56e931104610e6027a8ecabc0661527 Sep 16 00:47:42.669320 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:42.669299 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:47:42.990706 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:42.990668 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x" event={"ID":"8e2b07f9-744c-43d7-a97a-a6585938e0ee","Type":"ContainerStarted","Data":"940f565742670ec0db547139378f9e13b9f4d17b341c8a7558c752724b0ffd9c"} Sep 16 00:47:42.991003 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:42.990979 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x" Sep 16 00:47:42.991087 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:42.991026 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x" event={"ID":"8e2b07f9-744c-43d7-a97a-a6585938e0ee","Type":"ContainerStarted","Data":"6b2719a78e8f56ebafbe81f673fe6dd5f56e931104610e6027a8ecabc0661527"} Sep 16 00:47:43.008358 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:43.008314 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x" podStartSLOduration=2.008299875 podStartE2EDuration="2.008299875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:43.006759391 +0000 UTC m=+813.368982944" watchObservedRunningTime="2026-09-16 00:47:43.008299875 +0000 UTC m=+813.370523441" Sep 16 00:47:43.246023 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:43.245939 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:43.976954 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:43.976924 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:52.436820 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:52.436787 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:52.542539 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:52.542508 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:52.669434 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:52.669384 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:52.992942 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:52.992917 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg" Sep 16 00:47:53.837811 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:53.837775 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:47:55.003232 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:55.003202 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x" Sep 16 00:47:57.067463 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:57.067433 2567 generic.go:356] "Generic (PLEG): container finished" podID="3eff9577-aa6f-4e10-8538-ba0d09f132eb" containerID="ff74aa81986dfeb5bb051c9a0cf24d8dc0f4a116526a28b9e1d4481998ca664c" exitCode=0 Sep 16 00:47:57.067945 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:57.067474 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" event={"ID":"3eff9577-aa6f-4e10-8538-ba0d09f132eb","Type":"ContainerDied","Data":"ff74aa81986dfeb5bb051c9a0cf24d8dc0f4a116526a28b9e1d4481998ca664c"} Sep 16 00:47:57.067945 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:57.067909 2567 scope.go:117] "RemoveContainer" containerID="ff74aa81986dfeb5bb051c9a0cf24d8dc0f4a116526a28b9e1d4481998ca664c" Sep 16 00:47:57.068953 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:57.068927 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:47:57.069081 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:57.068992 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:47:57.069081 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:57.069021 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:47:58.075168 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:58.075135 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" event={"ID":"3eff9577-aa6f-4e10-8538-ba0d09f132eb","Type":"ContainerStarted","Data":"32a408ba2e0fe12373d9cdb74bbe42e7fc8a1a7b119c421186e0d24d7a6d068f"} Sep 16 00:47:58.368449 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:58.368360 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:58.412993 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:58.412965 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:47:58.413224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:58.413195 2567 patch_prober.go:28] interesting pod/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 16 00:47:58.413327 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:58.413250 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" podUID="3eff9577-aa6f-4e10-8538-ba0d09f132eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 16 00:47:58.477239 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:58.477207 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:58.568471 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:58.568412 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:58.724951 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:58.724869 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:47:59.413429 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:59.413390 2567 patch_prober.go:28] interesting pod/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 16 00:47:59.413817 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:47:59.413463 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" podUID="3eff9577-aa6f-4e10-8538-ba0d09f132eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 16 00:48:00.271851 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:00.271815 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:00.412855 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:00.412820 2567 patch_prober.go:28] interesting pod/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 16 00:48:00.413025 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:00.412873 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" podUID="3eff9577-aa6f-4e10-8538-ba0d09f132eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 16 00:48:01.413091 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:01.413024 2567 patch_prober.go:28] interesting pod/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 16 00:48:01.413500 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:01.413116 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" podUID="3eff9577-aa6f-4e10-8538-ba0d09f132eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 16 00:48:02.016341 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:02.016310 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:02.413011 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:02.412978 2567 patch_prober.go:28] interesting pod/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 16 00:48:02.413203 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:02.413032 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" podUID="3eff9577-aa6f-4e10-8538-ba0d09f132eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 16 00:48:02.697731 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:02.697649 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:03.036100 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.036014 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:03.140698 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.140651 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:03.314649 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.314611 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l"] Sep 16 00:48:03.319054 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.319016 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.322058 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.322019 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4d413f-openshift-default-dockercfg-x4qqd\"" Sep 16 00:48:03.332214 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.332189 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l"] Sep 16 00:48:03.339364 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.339339 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:03.429027 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.428466 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:48:03.429027 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.428873 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:48:03.429618 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.429601 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:48:03.464643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.464614 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-istio-data\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.464769 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.464654 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43f48e12-f207-480b-99c3-51eea6a1a852-istiod-ca-cert\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.464769 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.464674 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-istio-envoy\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.464849 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.464764 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-credential-socket\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.464849 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.464821 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-workload-certs\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.464918 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.464863 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6lxl\" (UniqueName: \"kubernetes.io/projected/43f48e12-f207-480b-99c3-51eea6a1a852-kube-api-access-v6lxl\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.464918 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.464881 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43f48e12-f207-480b-99c3-51eea6a1a852-istio-podinfo\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.464918 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.464915 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-workload-socket\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.465016 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.464933 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43f48e12-f207-480b-99c3-51eea6a1a852-istio-token\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.565920 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.565839 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6lxl\" (UniqueName: \"kubernetes.io/projected/43f48e12-f207-480b-99c3-51eea6a1a852-kube-api-access-v6lxl\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.565920 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.565878 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43f48e12-f207-480b-99c3-51eea6a1a852-istio-podinfo\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.566168 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.565985 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-workload-socket\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.566168 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.566023 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43f48e12-f207-480b-99c3-51eea6a1a852-istio-token\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.566168 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.566070 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-istio-data\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.566168 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.566126 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43f48e12-f207-480b-99c3-51eea6a1a852-istiod-ca-cert\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.566168 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.566154 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-istio-envoy\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.567029 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.566249 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-credential-socket\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.567029 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.566337 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-workload-certs\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.567029 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.566470 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-workload-socket\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.567029 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.566536 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-istio-data\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.567029 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.566784 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-credential-socket\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.567029 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.566886 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-workload-certs\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.567394 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.567173 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43f48e12-f207-480b-99c3-51eea6a1a852-istiod-ca-cert\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.568651 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.568629 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-istio-envoy\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.568752 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.568728 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43f48e12-f207-480b-99c3-51eea6a1a852-istio-podinfo\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.573778 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.573750 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43f48e12-f207-480b-99c3-51eea6a1a852-istio-token\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.573972 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.573957 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6lxl\" (UniqueName: \"kubernetes.io/projected/43f48e12-f207-480b-99c3-51eea6a1a852-kube-api-access-v6lxl\") pod \"e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.635299 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.635267 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:03.661545 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.661516 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:03.915819 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.915789 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l"] Sep 16 00:48:03.916893 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:48:03.916858 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f48e12_f207_480b_99c3_51eea6a1a852.slice/crio-5dcdddbf19bc98bfc10ca4c9bd50b55aadfb483a324e53de0a7ceda2828a7462 WatchSource:0}: Error finding container 5dcdddbf19bc98bfc10ca4c9bd50b55aadfb483a324e53de0a7ceda2828a7462: Status 404 returned error can't find the container with id 5dcdddbf19bc98bfc10ca4c9bd50b55aadfb483a324e53de0a7ceda2828a7462 Sep 16 00:48:03.919121 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.919088 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:48:03.919199 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.919154 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:48:03.919199 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:03.919183 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:48:04.052215 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:04.052186 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:04.109878 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:04.109840 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" event={"ID":"43f48e12-f207-480b-99c3-51eea6a1a852","Type":"ContainerStarted","Data":"2efa8462e03e07e86534304fde60276d34e90fcae0f1d50476d73e6bfa4f91b7"} Sep 16 00:48:04.110076 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:04.109907 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" event={"ID":"43f48e12-f207-480b-99c3-51eea6a1a852","Type":"ContainerStarted","Data":"5dcdddbf19bc98bfc10ca4c9bd50b55aadfb483a324e53de0a7ceda2828a7462"} Sep 16 00:48:04.130096 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:04.130029 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" podStartSLOduration=1.130014988 podStartE2EDuration="1.130014988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:48:04.128218803 +0000 UTC m=+834.490442360" watchObservedRunningTime="2026-09-16 00:48:04.130014988 +0000 UTC m=+834.492238519" Sep 16 00:48:04.636361 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:04.636323 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:04.641438 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:04.641415 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:04.808394 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:04.808356 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:05.115743 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:05.115712 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:05.116284 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:05.116268 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:05.173936 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:05.173907 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:05.273871 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:05.273839 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:05.837426 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:05.837379 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:06.114439 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:06.114323 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:06.210355 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:06.210321 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:06.570253 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:06.570228 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:07.409696 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:07.409659 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:07.509403 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:07.509364 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:07.609701 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:07.609667 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:07.867457 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:07.867417 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:08.041701 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:08.041666 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:08.073954 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:08.073923 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628"] Sep 16 00:48:08.077356 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:08.077340 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628" Sep 16 00:48:08.080106 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:08.080082 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4d413f\"" Sep 16 00:48:08.080203 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:08.080101 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4d413f-dockercfg-p4mbn\"" Sep 16 00:48:08.088250 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:08.088230 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628"] Sep 16 00:48:08.128760 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:08.128694 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:08.213952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:08.213917 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnwfp\" (UniqueName: \"kubernetes.io/projected/b4fd1ad7-62dc-46f0-9069-afe0a5f45e70-kube-api-access-qnwfp\") pod \"payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628\" (UID: \"b4fd1ad7-62dc-46f0-9069-afe0a5f45e70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628" Sep 16 00:48:08.213952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:08.213954 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4fd1ad7-62dc-46f0-9069-afe0a5f45e70-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628\" (UID: \"b4fd1ad7-62dc-46f0-9069-afe0a5f45e70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628" Sep 16 00:48:08.315354 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:08.315328 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnwfp\" (UniqueName: \"kubernetes.io/projected/b4fd1ad7-62dc-46f0-9069-afe0a5f45e70-kube-api-access-qnwfp\") pod \"payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628\" (UID: \"b4fd1ad7-62dc-46f0-9069-afe0a5f45e70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628" Sep 16 00:48:08.315514 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:08.315362 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4fd1ad7-62dc-46f0-9069-afe0a5f45e70-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628\" (UID: \"b4fd1ad7-62dc-46f0-9069-afe0a5f45e70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628" Sep 16 00:48:08.316023 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:08.316006 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4fd1ad7-62dc-46f0-9069-afe0a5f45e70-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628\" (UID: \"b4fd1ad7-62dc-46f0-9069-afe0a5f45e70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628" Sep 16 00:48:08.324718 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:08.324696 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnwfp\" (UniqueName: \"kubernetes.io/projected/b4fd1ad7-62dc-46f0-9069-afe0a5f45e70-kube-api-access-qnwfp\") pod \"payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628\" (UID: \"b4fd1ad7-62dc-46f0-9069-afe0a5f45e70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628" Sep 16 00:48:08.340246 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:08.340219 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:08.389456 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:08.389384 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628" Sep 16 00:48:08.581932 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:08.581814 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:08.668307 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:08.668272 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628"] Sep 16 00:48:08.669454 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:48:08.669425 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fd1ad7_62dc_46f0_9069_afe0a5f45e70.slice/crio-e2f5f2b4c3bf137ce2c326ae1c95db65b743b576dfb93fb6160cbe71f6308c20 WatchSource:0}: Error finding container e2f5f2b4c3bf137ce2c326ae1c95db65b743b576dfb93fb6160cbe71f6308c20: Status 404 returned error can't find the container with id e2f5f2b4c3bf137ce2c326ae1c95db65b743b576dfb93fb6160cbe71f6308c20 Sep 16 00:48:08.669632 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:08.669495 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:08.736925 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:08.736888 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:08.837209 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:08.837178 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:09.135813 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:09.135777 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:09.138527 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:09.138502 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628" event={"ID":"b4fd1ad7-62dc-46f0-9069-afe0a5f45e70","Type":"ContainerStarted","Data":"645dcc09cfee83a00248287952ce36a8d93e7c640a5bbdde7d29c7aa6543e342"} Sep 16 00:48:09.138646 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:09.138572 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628" event={"ID":"b4fd1ad7-62dc-46f0-9069-afe0a5f45e70","Type":"ContainerStarted","Data":"e2f5f2b4c3bf137ce2c326ae1c95db65b743b576dfb93fb6160cbe71f6308c20"} Sep 16 00:48:09.138646 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:09.138614 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628" Sep 16 00:48:09.159366 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:09.159323 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628" podStartSLOduration=1.159307251 podStartE2EDuration="1.159307251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:48:09.156634236 +0000 UTC m=+839.518857782" watchObservedRunningTime="2026-09-16 00:48:09.159307251 +0000 UTC m=+839.521530786" Sep 16 00:48:09.254443 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:09.254295 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:09.348572 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:09.348546 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:09.450797 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:09.450713 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:12.725821 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:12.725790 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:13.034237 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:13.034154 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:48:19.991989 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:19.991900 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz"] Sep 16 00:48:19.997647 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:19.997624 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:20.001741 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.001717 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-0f7c3b\"/\"default-dockercfg-trr95\"" Sep 16 00:48:20.002097 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.001956 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-0f7c3b\"/\"openshift-service-ca.crt\"" Sep 16 00:48:20.002097 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.001967 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-0f7c3b\"/\"kube-root-ca.crt\"" Sep 16 00:48:20.002593 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.002489 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-0f7c3b\"/\"simulator-w3-0f7c3b-kserve-self-signed-certs\"" Sep 16 00:48:20.005009 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.004991 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz"] Sep 16 00:48:20.018321 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.018297 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:20.124853 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.124808 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-tls-certs\") pod \"simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:20.125197 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.125165 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhsvk\" (UniqueName: \"kubernetes.io/projected/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-kube-api-access-dhsvk\") pod \"simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:20.125343 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.125240 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-model-cache\") pod \"simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:20.125395 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.125345 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-home\") pod \"simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:20.125395 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.125378 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-tmp-dir\") pod \"simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:20.125487 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.125421 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-dshm\") pod \"simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:20.145315 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.145287 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628" Sep 16 00:48:20.227360 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.227330 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-dshm\") pod \"simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:20.227539 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.227434 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-tls-certs\") pod \"simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:20.227539 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.227491 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhsvk\" (UniqueName: \"kubernetes.io/projected/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-kube-api-access-dhsvk\") pod \"simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:20.227662 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.227538 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-model-cache\") pod \"simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:20.227662 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.227579 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-home\") pod \"simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:20.227662 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.227602 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-tmp-dir\") pod \"simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:20.228009 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.227952 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-model-cache\") pod \"simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:20.228160 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.228076 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-tmp-dir\") pod \"simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:20.228160 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.228103 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-home\") pod \"simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:20.230427 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.230404 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-tls-certs\") pod \"simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:20.231006 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.230987 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-dshm\") pod \"simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:20.235554 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.235528 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhsvk\" (UniqueName: \"kubernetes.io/projected/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-kube-api-access-dhsvk\") pod \"simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:20.311801 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.311765 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:20.591437 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:20.591406 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz"] Sep 16 00:48:20.593119 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:48:20.593092 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddbbcc45_8ae7_4595_bc7c_257ef711cee9.slice/crio-ea41e9954b1124fb093e07329e044699b69272e1fa2bb0667f278ae2f3678467 WatchSource:0}: Error finding container ea41e9954b1124fb093e07329e044699b69272e1fa2bb0667f278ae2f3678467: Status 404 returned error can't find the container with id ea41e9954b1124fb093e07329e044699b69272e1fa2bb0667f278ae2f3678467 Sep 16 00:48:21.156201 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:21.156162 2567 patch_prober.go:28] interesting pod/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 16 00:48:21.156637 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:21.156239 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" podUID="43f48e12-f207-480b-99c3-51eea6a1a852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 16 00:48:21.213540 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:21.213501 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" event={"ID":"ddbbcc45-8ae7-4595-bc7c-257ef711cee9","Type":"ContainerStarted","Data":"43f7f7cdf93f99db33c6ed76097f171c280662c225cc611c53245c8ae08d7806"} Sep 16 00:48:21.213710 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:21.213548 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" event={"ID":"ddbbcc45-8ae7-4595-bc7c-257ef711cee9","Type":"ContainerStarted","Data":"ea41e9954b1124fb093e07329e044699b69272e1fa2bb0667f278ae2f3678467"} Sep 16 00:48:21.231362 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:21.231305 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" podStartSLOduration=2.231288383 podStartE2EDuration="2.231288383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:48:21.229322079 +0000 UTC m=+851.591545632" watchObservedRunningTime="2026-09-16 00:48:21.231288383 +0000 UTC m=+851.593511916" Sep 16 00:48:23.226784 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:23.226753 2567 generic.go:356] "Generic (PLEG): container finished" podID="43f48e12-f207-480b-99c3-51eea6a1a852" containerID="2efa8462e03e07e86534304fde60276d34e90fcae0f1d50476d73e6bfa4f91b7" exitCode=0 Sep 16 00:48:23.227238 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:23.226840 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" event={"ID":"43f48e12-f207-480b-99c3-51eea6a1a852","Type":"ContainerDied","Data":"2efa8462e03e07e86534304fde60276d34e90fcae0f1d50476d73e6bfa4f91b7"} Sep 16 00:48:23.227286 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:23.227258 2567 scope.go:117] "RemoveContainer" containerID="2efa8462e03e07e86534304fde60276d34e90fcae0f1d50476d73e6bfa4f91b7" Sep 16 00:48:23.228274 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:23.228245 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:48:23.228383 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:23.228308 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:48:23.228383 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:23.228340 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:48:24.240294 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:24.240255 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" event={"ID":"43f48e12-f207-480b-99c3-51eea6a1a852","Type":"ContainerStarted","Data":"4d3303125015a673f89b8b7306d11ceb9d4d030f69ca73615be3b85d5b5ad467"} Sep 16 00:48:24.635822 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:24.635789 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:24.635998 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:24.635964 2567 patch_prober.go:28] interesting pod/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Sep 16 00:48:24.636070 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:24.636007 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" podUID="43f48e12-f207-480b-99c3-51eea6a1a852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Sep 16 00:48:25.636279 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:25.636245 2567 patch_prober.go:28] interesting pod/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Sep 16 00:48:25.636663 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:25.636312 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" podUID="43f48e12-f207-480b-99c3-51eea6a1a852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Sep 16 00:48:26.636274 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:26.636240 2567 patch_prober.go:28] interesting pod/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Sep 16 00:48:26.636448 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:26.636300 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" podUID="43f48e12-f207-480b-99c3-51eea6a1a852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Sep 16 00:48:26.748824 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:26.748790 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:48:27.636068 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:27.636015 2567 patch_prober.go:28] interesting pod/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Sep 16 00:48:27.636267 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:27.636160 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" podUID="43f48e12-f207-480b-99c3-51eea6a1a852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Sep 16 00:48:28.636456 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:28.636416 2567 patch_prober.go:28] interesting pod/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Sep 16 00:48:28.636833 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:28.636472 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" podUID="43f48e12-f207-480b-99c3-51eea6a1a852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Sep 16 00:48:29.308053 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:29.308009 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:29.757470 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:29.757370 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:29.758082 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:29.758024 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:29.758799 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:29.758779 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:48:30.148675 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:30.148647 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:30.312567 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:30.312540 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:30.312824 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:30.312802 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:30.324731 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:30.324710 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:30.799658 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:30.799627 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:31.130197 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:31.130171 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:31.307704 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:31.307669 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:48:31.734286 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:31.734251 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:47.523865 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:47.523807 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:48:52.638954 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:52.638904 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:48:58.887851 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:48:58.887818 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:49:02.396662 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.396571 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq"] Sep 16 00:49:02.400667 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.400640 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:02.403355 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.403326 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-0f7c3b\"/\"premium-w3-0f7c3b-kserve-self-signed-certs\"" Sep 16 00:49:02.412605 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.412576 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq"] Sep 16 00:49:02.434128 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.434091 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:49:02.437076 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.435511 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-tmp-dir\") pod \"premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:02.437076 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.435582 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd5906f5-99bd-4608-9e84-19385f7daf27-tls-certs\") pod \"premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:02.437076 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.435614 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-home\") pod \"premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:02.437076 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.435726 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98lrf\" (UniqueName: \"kubernetes.io/projected/bd5906f5-99bd-4608-9e84-19385f7daf27-kube-api-access-98lrf\") pod \"premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:02.437076 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.435758 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-dshm\") pod \"premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:02.437076 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.435810 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-model-cache\") pod \"premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:02.537349 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.537299 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98lrf\" (UniqueName: \"kubernetes.io/projected/bd5906f5-99bd-4608-9e84-19385f7daf27-kube-api-access-98lrf\") pod \"premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:02.537349 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.537350 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-dshm\") pod \"premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:02.537636 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.537378 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-model-cache\") pod \"premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:02.537636 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.537428 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-tmp-dir\") pod \"premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:02.537636 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.537449 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd5906f5-99bd-4608-9e84-19385f7daf27-tls-certs\") pod \"premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:02.537636 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.537464 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-home\") pod \"premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:02.537953 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.537927 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-tmp-dir\") pod \"premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:02.538072 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.537953 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-model-cache\") pod \"premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:02.538072 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.537979 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-home\") pod \"premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:02.540110 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.540054 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-dshm\") pod \"premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:02.540239 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.540221 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd5906f5-99bd-4608-9e84-19385f7daf27-tls-certs\") pod \"premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:02.546967 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.546937 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98lrf\" (UniqueName: \"kubernetes.io/projected/bd5906f5-99bd-4608-9e84-19385f7daf27-kube-api-access-98lrf\") pod \"premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:02.736258 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:02.736153 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:03.013082 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:03.013026 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq"] Sep 16 00:49:03.015468 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:49:03.015435 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd5906f5_99bd_4608_9e84_19385f7daf27.slice/crio-c3528560f0bfb0c51de6f8455ee1f3602dec7f24fad59d17ea72ae99a9d6308f WatchSource:0}: Error finding container c3528560f0bfb0c51de6f8455ee1f3602dec7f24fad59d17ea72ae99a9d6308f: Status 404 returned error can't find the container with id c3528560f0bfb0c51de6f8455ee1f3602dec7f24fad59d17ea72ae99a9d6308f Sep 16 00:49:03.485371 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:03.485329 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" event={"ID":"bd5906f5-99bd-4608-9e84-19385f7daf27","Type":"ContainerStarted","Data":"951e2f23607bc6c6ed97170c96137e9d8ae53ad763c919c6f2d0805be9bf80cc"} Sep 16 00:49:03.485371 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:03.485380 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" event={"ID":"bd5906f5-99bd-4608-9e84-19385f7daf27","Type":"ContainerStarted","Data":"c3528560f0bfb0c51de6f8455ee1f3602dec7f24fad59d17ea72ae99a9d6308f"} Sep 16 00:49:03.506818 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:03.506757 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" podStartSLOduration=1.506739771 podStartE2EDuration="1.506739771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:49:03.503474483 +0000 UTC m=+893.865698037" watchObservedRunningTime="2026-09-16 00:49:03.506739771 +0000 UTC m=+893.868963349" Sep 16 00:49:12.736539 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:12.736491 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:12.736932 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:12.736567 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:12.749793 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:12.749746 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:13.562459 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:13.562429 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:49:13.846447 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:13.846354 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:49:38.435792 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:38.435740 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:49:40.817692 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:40.817655 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:49:42.154741 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:42.154701 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:49:43.790345 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.790308 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq"] Sep 16 00:49:43.794005 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.793981 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:43.798139 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.798110 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4d413f\"/\"default-dockercfg-nwt55\"" Sep 16 00:49:43.798307 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.798141 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4d413f\"/\"openshift-service-ca.crt\"" Sep 16 00:49:43.798307 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.798098 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4d413f\"/\"kube-root-ca.crt\"" Sep 16 00:49:43.798307 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.798116 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4d413f\"/\"test-model-4d413f-kserve-self-signed-certs\"" Sep 16 00:49:43.805705 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.805669 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq"] Sep 16 00:49:43.808313 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.808275 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:49:43.828884 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.828838 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-home\") pod \"test-model-4d413f-kserve-d8cfcf6d9-zqszq\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:43.829165 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.828907 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5b92ac-00d0-45ca-a614-c1e662532f97-tls-certs\") pod \"test-model-4d413f-kserve-d8cfcf6d9-zqszq\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:43.829165 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.828958 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-model-cache\") pod \"test-model-4d413f-kserve-d8cfcf6d9-zqszq\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:43.829165 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.828985 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wztb7\" (UniqueName: \"kubernetes.io/projected/eb5b92ac-00d0-45ca-a614-c1e662532f97-kube-api-access-wztb7\") pod \"test-model-4d413f-kserve-d8cfcf6d9-zqszq\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:43.829165 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.829013 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-tmp-dir\") pod \"test-model-4d413f-kserve-d8cfcf6d9-zqszq\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:43.829165 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.829103 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-dshm\") pod \"test-model-4d413f-kserve-d8cfcf6d9-zqszq\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:43.929639 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.929591 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-home\") pod \"test-model-4d413f-kserve-d8cfcf6d9-zqszq\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:43.929955 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.929682 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5b92ac-00d0-45ca-a614-c1e662532f97-tls-certs\") pod \"test-model-4d413f-kserve-d8cfcf6d9-zqszq\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:43.929955 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.929727 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-model-cache\") pod \"test-model-4d413f-kserve-d8cfcf6d9-zqszq\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:43.929955 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.929757 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wztb7\" (UniqueName: \"kubernetes.io/projected/eb5b92ac-00d0-45ca-a614-c1e662532f97-kube-api-access-wztb7\") pod \"test-model-4d413f-kserve-d8cfcf6d9-zqszq\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:43.929955 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.929788 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-tmp-dir\") pod \"test-model-4d413f-kserve-d8cfcf6d9-zqszq\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:43.929955 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.929824 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-dshm\") pod \"test-model-4d413f-kserve-d8cfcf6d9-zqszq\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:43.930304 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.930078 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-home\") pod \"test-model-4d413f-kserve-d8cfcf6d9-zqszq\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:43.930304 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.930139 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-model-cache\") pod \"test-model-4d413f-kserve-d8cfcf6d9-zqszq\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:43.930304 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.930190 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-tmp-dir\") pod \"test-model-4d413f-kserve-d8cfcf6d9-zqszq\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:43.932322 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.932299 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-dshm\") pod \"test-model-4d413f-kserve-d8cfcf6d9-zqszq\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:43.932485 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.932468 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5b92ac-00d0-45ca-a614-c1e662532f97-tls-certs\") pod \"test-model-4d413f-kserve-d8cfcf6d9-zqszq\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:43.938596 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:43.938566 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wztb7\" (UniqueName: \"kubernetes.io/projected/eb5b92ac-00d0-45ca-a614-c1e662532f97-kube-api-access-wztb7\") pod \"test-model-4d413f-kserve-d8cfcf6d9-zqszq\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:44.108686 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:44.108643 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:44.386640 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:44.386610 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq"] Sep 16 00:49:44.388127 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:49:44.388084 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb5b92ac_00d0_45ca_a614_c1e662532f97.slice/crio-23a0722304b3ee5708e276e408cf9808bd6ca029a9e1023f77e1398da98f189e WatchSource:0}: Error finding container 23a0722304b3ee5708e276e408cf9808bd6ca029a9e1023f77e1398da98f189e: Status 404 returned error can't find the container with id 23a0722304b3ee5708e276e408cf9808bd6ca029a9e1023f77e1398da98f189e Sep 16 00:49:44.732441 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:44.732339 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" event={"ID":"eb5b92ac-00d0-45ca-a614-c1e662532f97","Type":"ContainerStarted","Data":"65212bb8b26f2669a282d5f81db7c4fce5651a93ef31150e16a23175b9d9c68f"} Sep 16 00:49:44.732441 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:44.732391 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" event={"ID":"eb5b92ac-00d0-45ca-a614-c1e662532f97","Type":"ContainerStarted","Data":"23a0722304b3ee5708e276e408cf9808bd6ca029a9e1023f77e1398da98f189e"} Sep 16 00:49:44.751935 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:44.751873 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" podStartSLOduration=1.7518523240000001 podStartE2EDuration="1.751852324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:49:44.749688941 +0000 UTC m=+935.111912483" watchObservedRunningTime="2026-09-16 00:49:44.751852324 +0000 UTC m=+935.114075856" Sep 16 00:49:52.844879 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:52.844837 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:49:54.109730 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:54.109687 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:54.110192 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:54.109770 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:54.123743 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:54.123712 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:49:54.801915 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:49:54.801880 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:50:06.423068 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:06.423013 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:50:21.740158 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:21.740119 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:50:25.580764 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.580728 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv"] Sep 16 00:50:25.584352 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.584332 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:25.587014 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.586989 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5270e2\"/\"kube-root-ca.crt\"" Sep 16 00:50:25.591060 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.587604 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5270e2\"/\"openshift-service-ca.crt\"" Sep 16 00:50:25.594410 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.594378 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5270e2\"/\"default-dockercfg-fmb5k\"" Sep 16 00:50:25.595152 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.595126 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5270e2\"/\"distinct-w0-5270e2-kserve-self-signed-certs\"" Sep 16 00:50:25.598672 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.598639 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv"] Sep 16 00:50:25.639109 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.639069 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ltcx\" (UniqueName: \"kubernetes.io/projected/c164b60c-870c-42bc-be7f-59f517de59f2-kube-api-access-6ltcx\") pod \"distinct-w0-5270e2-kserve-5cdc6db996-62kjv\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:25.639331 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.639137 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-model-cache\") pod \"distinct-w0-5270e2-kserve-5cdc6db996-62kjv\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:25.639331 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.639203 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-tmp-dir\") pod \"distinct-w0-5270e2-kserve-5cdc6db996-62kjv\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:25.639331 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.639235 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c164b60c-870c-42bc-be7f-59f517de59f2-tls-certs\") pod \"distinct-w0-5270e2-kserve-5cdc6db996-62kjv\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:25.639331 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.639255 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-home\") pod \"distinct-w0-5270e2-kserve-5cdc6db996-62kjv\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:25.639331 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.639297 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-dshm\") pod \"distinct-w0-5270e2-kserve-5cdc6db996-62kjv\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:25.740314 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.740277 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ltcx\" (UniqueName: \"kubernetes.io/projected/c164b60c-870c-42bc-be7f-59f517de59f2-kube-api-access-6ltcx\") pod \"distinct-w0-5270e2-kserve-5cdc6db996-62kjv\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:25.740547 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.740355 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-model-cache\") pod \"distinct-w0-5270e2-kserve-5cdc6db996-62kjv\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:25.740547 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.740408 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-tmp-dir\") pod \"distinct-w0-5270e2-kserve-5cdc6db996-62kjv\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:25.740547 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.740449 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c164b60c-870c-42bc-be7f-59f517de59f2-tls-certs\") pod \"distinct-w0-5270e2-kserve-5cdc6db996-62kjv\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:25.740547 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.740476 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-home\") pod \"distinct-w0-5270e2-kserve-5cdc6db996-62kjv\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:25.740547 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.740537 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-dshm\") pod \"distinct-w0-5270e2-kserve-5cdc6db996-62kjv\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:25.740844 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.740821 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-model-cache\") pod \"distinct-w0-5270e2-kserve-5cdc6db996-62kjv\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:25.740910 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.740865 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-tmp-dir\") pod \"distinct-w0-5270e2-kserve-5cdc6db996-62kjv\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:25.740910 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.740887 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-home\") pod \"distinct-w0-5270e2-kserve-5cdc6db996-62kjv\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:25.743011 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.742988 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-dshm\") pod \"distinct-w0-5270e2-kserve-5cdc6db996-62kjv\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:25.743290 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.743271 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c164b60c-870c-42bc-be7f-59f517de59f2-tls-certs\") pod \"distinct-w0-5270e2-kserve-5cdc6db996-62kjv\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:25.748660 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.748629 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ltcx\" (UniqueName: \"kubernetes.io/projected/c164b60c-870c-42bc-be7f-59f517de59f2-kube-api-access-6ltcx\") pod \"distinct-w0-5270e2-kserve-5cdc6db996-62kjv\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:25.786878 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.786845 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:50:25.902759 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:25.902654 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:26.185921 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:26.185881 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv"] Sep 16 00:50:26.186759 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:50:26.186729 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc164b60c_870c_42bc_be7f_59f517de59f2.slice/crio-7b2e86b9f8d8c0d093c62a03ffddc0a017d7790f292c4721bce9b506500a85c2 WatchSource:0}: Error finding container 7b2e86b9f8d8c0d093c62a03ffddc0a017d7790f292c4721bce9b506500a85c2: Status 404 returned error can't find the container with id 7b2e86b9f8d8c0d093c62a03ffddc0a017d7790f292c4721bce9b506500a85c2 Sep 16 00:50:26.987859 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:26.987817 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" event={"ID":"c164b60c-870c-42bc-be7f-59f517de59f2","Type":"ContainerStarted","Data":"2648d37329e185d9a5b02637c0ad439e277c260b448755b0df79272e2fd81b7e"} Sep 16 00:50:26.987859 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:26.987860 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" event={"ID":"c164b60c-870c-42bc-be7f-59f517de59f2","Type":"ContainerStarted","Data":"7b2e86b9f8d8c0d093c62a03ffddc0a017d7790f292c4721bce9b506500a85c2"} Sep 16 00:50:27.005116 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:27.005055 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" podStartSLOduration=2.005008456 podStartE2EDuration="2.005008456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:50:27.002807724 +0000 UTC m=+977.365031257" watchObservedRunningTime="2026-09-16 00:50:27.005008456 +0000 UTC m=+977.367231987" Sep 16 00:50:31.569860 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:31.569778 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:50:35.902895 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:35.902850 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:35.903372 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:35.902924 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:35.915848 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:35.915818 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:36.053249 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:36.053215 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:50:42.524483 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:42.524435 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:50:48.637533 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:48.637488 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:50:49.381273 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:49.381234 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:50:50.435768 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:50.435727 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:50:50.537574 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:50.537543 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:50:59.598074 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:50:59.598020 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:05.424304 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:05.424263 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:08.237620 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:08.237582 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq"] Sep 16 00:51:08.252300 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:08.252262 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:08.363820 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:08.363784 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ecb7-6c7cv"] Sep 16 00:51:08.369145 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:08.369081 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ecb7-6c7cv" Sep 16 00:51:08.374415 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:08.374379 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 00:51:08.374609 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:08.374505 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-npltg\"" Sep 16 00:51:08.376795 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:08.376766 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ecb7-6c7cv"] Sep 16 00:51:08.466716 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:08.466668 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d35824b-1a22-4c15-b39a-23d9e955d35a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-66ecb7-6c7cv\" (UID: \"4d35824b-1a22-4c15-b39a-23d9e955d35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ecb7-6c7cv" Sep 16 00:51:08.567936 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:08.567898 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d35824b-1a22-4c15-b39a-23d9e955d35a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-66ecb7-6c7cv\" (UID: \"4d35824b-1a22-4c15-b39a-23d9e955d35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ecb7-6c7cv" Sep 16 00:51:08.568605 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:08.568582 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d35824b-1a22-4c15-b39a-23d9e955d35a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-66ecb7-6c7cv\" (UID: \"4d35824b-1a22-4c15-b39a-23d9e955d35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ecb7-6c7cv" Sep 16 00:51:08.688410 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:08.688366 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ecb7-6c7cv" Sep 16 00:51:08.717622 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:08.717584 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:08.964221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:08.964196 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ecb7-6c7cv"] Sep 16 00:51:08.966026 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:51:08.965996 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d35824b_1a22_4c15_b39a_23d9e955d35a.slice/crio-15af7fa5c26efafe57c92ded0d2784624062724f7cdefb4881a690b50006f7a6 WatchSource:0}: Error finding container 15af7fa5c26efafe57c92ded0d2784624062724f7cdefb4881a690b50006f7a6: Status 404 returned error can't find the container with id 15af7fa5c26efafe57c92ded0d2784624062724f7cdefb4881a690b50006f7a6 Sep 16 00:51:09.257079 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:09.257028 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ecb7-6c7cv" event={"ID":"4d35824b-1a22-4c15-b39a-23d9e955d35a","Type":"ContainerStarted","Data":"15af7fa5c26efafe57c92ded0d2784624062724f7cdefb4881a690b50006f7a6"} Sep 16 00:51:09.515677 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:09.515564 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:10.152095 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:10.152065 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:10.264442 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:10.264402 2567 generic.go:356] "Generic (PLEG): container finished" podID="4d35824b-1a22-4c15-b39a-23d9e955d35a" containerID="84d3f8ea36b69ec9a942012e61e4d0fae5fd1dff57029d434480c3da018728da" exitCode=0 Sep 16 00:51:10.264917 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:10.264498 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ecb7-6c7cv" event={"ID":"4d35824b-1a22-4c15-b39a-23d9e955d35a","Type":"ContainerDied","Data":"84d3f8ea36b69ec9a942012e61e4d0fae5fd1dff57029d434480c3da018728da"} Sep 16 00:51:10.827829 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:10.827789 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:11.417000 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:11.416973 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ecb7-6c7cv" Sep 16 00:51:11.499072 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:11.498997 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d35824b-1a22-4c15-b39a-23d9e955d35a-maas-api-service-ca\") pod \"4d35824b-1a22-4c15-b39a-23d9e955d35a\" (UID: \"4d35824b-1a22-4c15-b39a-23d9e955d35a\") " Sep 16 00:51:11.499394 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:11.499370 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d35824b-1a22-4c15-b39a-23d9e955d35a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4d35824b-1a22-4c15-b39a-23d9e955d35a" (UID: "4d35824b-1a22-4c15-b39a-23d9e955d35a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:51:11.600121 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:11.600079 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d35824b-1a22-4c15-b39a-23d9e955d35a-maas-api-service-ca\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:11.734485 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:11.734448 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:12.277996 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:12.277967 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ecb7-6c7cv" Sep 16 00:51:12.278208 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:12.278008 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ecb7-6c7cv" event={"ID":"4d35824b-1a22-4c15-b39a-23d9e955d35a","Type":"ContainerDied","Data":"15af7fa5c26efafe57c92ded0d2784624062724f7cdefb4881a690b50006f7a6"} Sep 16 00:51:12.278208 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:12.278089 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15af7fa5c26efafe57c92ded0d2784624062724f7cdefb4881a690b50006f7a6" Sep 16 00:51:18.870138 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:18.870068 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:18.905349 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:18.905309 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg"] Sep 16 00:51:18.905580 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:18.905549 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg" podUID="fe6fcd3b-33cf-4706-af7f-54b182eb6821" containerName="payload-processing" containerID="cri-o://47d2f75cc1739b53d2d31551b21dc17396ed91f072946d686a6eb91700eee6f1" gracePeriod=30 Sep 16 00:51:18.970178 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:18.970128 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:19.066654 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.066580 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ecb7-6c7cv"] Sep 16 00:51:19.066863 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.066762 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:19.069418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.069384 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ecb7-6c7cv"] Sep 16 00:51:19.133985 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.133898 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:19.190474 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.190446 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg" Sep 16 00:51:19.280121 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.280076 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe6fcd3b-33cf-4706-af7f-54b182eb6821-plugins-config-volume\") pod \"fe6fcd3b-33cf-4706-af7f-54b182eb6821\" (UID: \"fe6fcd3b-33cf-4706-af7f-54b182eb6821\") " Sep 16 00:51:19.280301 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.280167 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vst4\" (UniqueName: \"kubernetes.io/projected/fe6fcd3b-33cf-4706-af7f-54b182eb6821-kube-api-access-6vst4\") pod \"fe6fcd3b-33cf-4706-af7f-54b182eb6821\" (UID: \"fe6fcd3b-33cf-4706-af7f-54b182eb6821\") " Sep 16 00:51:19.280479 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.280452 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe6fcd3b-33cf-4706-af7f-54b182eb6821-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe6fcd3b-33cf-4706-af7f-54b182eb6821" (UID: "fe6fcd3b-33cf-4706-af7f-54b182eb6821"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:51:19.282992 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.282958 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe6fcd3b-33cf-4706-af7f-54b182eb6821-kube-api-access-6vst4" (OuterVolumeSpecName: "kube-api-access-6vst4") pod "fe6fcd3b-33cf-4706-af7f-54b182eb6821" (UID: "fe6fcd3b-33cf-4706-af7f-54b182eb6821"). InnerVolumeSpecName "kube-api-access-6vst4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:51:19.321135 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.321083 2567 generic.go:356] "Generic (PLEG): container finished" podID="fe6fcd3b-33cf-4706-af7f-54b182eb6821" containerID="47d2f75cc1739b53d2d31551b21dc17396ed91f072946d686a6eb91700eee6f1" exitCode=0 Sep 16 00:51:19.321354 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.321200 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg" Sep 16 00:51:19.321354 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.321210 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg" event={"ID":"fe6fcd3b-33cf-4706-af7f-54b182eb6821","Type":"ContainerDied","Data":"47d2f75cc1739b53d2d31551b21dc17396ed91f072946d686a6eb91700eee6f1"} Sep 16 00:51:19.321354 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.321258 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg" event={"ID":"fe6fcd3b-33cf-4706-af7f-54b182eb6821","Type":"ContainerDied","Data":"1c04cd490acfc89520792bdee65429131720c8d617a9f6a0b139caf491d5e448"} Sep 16 00:51:19.321354 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.321280 2567 scope.go:117] "RemoveContainer" containerID="47d2f75cc1739b53d2d31551b21dc17396ed91f072946d686a6eb91700eee6f1" Sep 16 00:51:19.331646 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.331613 2567 scope.go:117] "RemoveContainer" containerID="47d2f75cc1739b53d2d31551b21dc17396ed91f072946d686a6eb91700eee6f1" Sep 16 00:51:19.332002 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:51:19.331959 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d2f75cc1739b53d2d31551b21dc17396ed91f072946d686a6eb91700eee6f1\": container with ID starting with 47d2f75cc1739b53d2d31551b21dc17396ed91f072946d686a6eb91700eee6f1 not found: ID does not exist" containerID="47d2f75cc1739b53d2d31551b21dc17396ed91f072946d686a6eb91700eee6f1" Sep 16 00:51:19.332116 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.332009 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d2f75cc1739b53d2d31551b21dc17396ed91f072946d686a6eb91700eee6f1"} err="failed to get container status \"47d2f75cc1739b53d2d31551b21dc17396ed91f072946d686a6eb91700eee6f1\": rpc error: code = NotFound desc = could not find container \"47d2f75cc1739b53d2d31551b21dc17396ed91f072946d686a6eb91700eee6f1\": container with ID starting with 47d2f75cc1739b53d2d31551b21dc17396ed91f072946d686a6eb91700eee6f1 not found: ID does not exist" Sep 16 00:51:19.347188 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.347141 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg"] Sep 16 00:51:19.349740 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.349714 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-66ecb7-6db59d5575-vl6pg"] Sep 16 00:51:19.381818 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.381779 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vst4\" (UniqueName: \"kubernetes.io/projected/fe6fcd3b-33cf-4706-af7f-54b182eb6821-kube-api-access-6vst4\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:19.381818 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.381814 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe6fcd3b-33cf-4706-af7f-54b182eb6821-plugins-config-volume\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:19.412106 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.411980 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:19.512464 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:19.512429 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:20.118786 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:20.118740 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d35824b-1a22-4c15-b39a-23d9e955d35a" path="/var/lib/kubelet/pods/4d35824b-1a22-4c15-b39a-23d9e955d35a/volumes" Sep 16 00:51:20.119343 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:20.119320 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe6fcd3b-33cf-4706-af7f-54b182eb6821" path="/var/lib/kubelet/pods/fe6fcd3b-33cf-4706-af7f-54b182eb6821/volumes" Sep 16 00:51:20.181628 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:20.181592 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:20.828796 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:20.828764 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:22.442464 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:22.442407 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:51:23.261623 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:23.261574 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" podUID="eb5b92ac-00d0-45ca-a614-c1e662532f97" containerName="main" containerID="cri-o://65212bb8b26f2669a282d5f81db7c4fce5651a93ef31150e16a23175b9d9c68f" gracePeriod=45 Sep 16 00:51:23.555571 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:23.555535 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:51:23.626715 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:23.626658 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wztb7\" (UniqueName: \"kubernetes.io/projected/eb5b92ac-00d0-45ca-a614-c1e662532f97-kube-api-access-wztb7\") pod \"eb5b92ac-00d0-45ca-a614-c1e662532f97\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " Sep 16 00:51:23.626715 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:23.626711 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-tmp-dir\") pod \"eb5b92ac-00d0-45ca-a614-c1e662532f97\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " Sep 16 00:51:23.626988 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:23.626759 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-dshm\") pod \"eb5b92ac-00d0-45ca-a614-c1e662532f97\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " Sep 16 00:51:23.626988 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:23.626801 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-model-cache\") pod \"eb5b92ac-00d0-45ca-a614-c1e662532f97\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " Sep 16 00:51:23.626988 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:23.626894 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-home\") pod \"eb5b92ac-00d0-45ca-a614-c1e662532f97\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " Sep 16 00:51:23.626988 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:23.626923 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5b92ac-00d0-45ca-a614-c1e662532f97-tls-certs\") pod \"eb5b92ac-00d0-45ca-a614-c1e662532f97\" (UID: \"eb5b92ac-00d0-45ca-a614-c1e662532f97\") " Sep 16 00:51:23.627242 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:23.627086 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb5b92ac-00d0-45ca-a614-c1e662532f97" (UID: "eb5b92ac-00d0-45ca-a614-c1e662532f97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:51:23.627242 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:23.627094 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb5b92ac-00d0-45ca-a614-c1e662532f97" (UID: "eb5b92ac-00d0-45ca-a614-c1e662532f97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:51:23.627580 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:23.627438 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-home" (OuterVolumeSpecName: "home") pod "eb5b92ac-00d0-45ca-a614-c1e662532f97" (UID: "eb5b92ac-00d0-45ca-a614-c1e662532f97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:51:23.627580 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:23.627454 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-tmp-dir\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:23.627580 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:23.627478 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-model-cache\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:23.630073 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:23.629974 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-dshm" (OuterVolumeSpecName: "dshm") pod "eb5b92ac-00d0-45ca-a614-c1e662532f97" (UID: "eb5b92ac-00d0-45ca-a614-c1e662532f97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:51:23.630073 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:23.630007 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb5b92ac-00d0-45ca-a614-c1e662532f97-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb5b92ac-00d0-45ca-a614-c1e662532f97" (UID: "eb5b92ac-00d0-45ca-a614-c1e662532f97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 00:51:23.630279 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:23.630183 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb5b92ac-00d0-45ca-a614-c1e662532f97-kube-api-access-wztb7" (OuterVolumeSpecName: "kube-api-access-wztb7") pod "eb5b92ac-00d0-45ca-a614-c1e662532f97" (UID: "eb5b92ac-00d0-45ca-a614-c1e662532f97"). InnerVolumeSpecName "kube-api-access-wztb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:51:23.729005 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:23.728955 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wztb7\" (UniqueName: \"kubernetes.io/projected/eb5b92ac-00d0-45ca-a614-c1e662532f97-kube-api-access-wztb7\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:23.729005 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:23.728995 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-dshm\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:23.729005 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:23.729005 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb5b92ac-00d0-45ca-a614-c1e662532f97-home\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:23.729005 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:23.729024 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5b92ac-00d0-45ca-a614-c1e662532f97-tls-certs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:24.362597 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:24.362560 2567 generic.go:356] "Generic (PLEG): container finished" podID="eb5b92ac-00d0-45ca-a614-c1e662532f97" containerID="65212bb8b26f2669a282d5f81db7c4fce5651a93ef31150e16a23175b9d9c68f" exitCode=0 Sep 16 00:51:24.362814 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:24.362620 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" Sep 16 00:51:24.362814 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:24.362653 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" event={"ID":"eb5b92ac-00d0-45ca-a614-c1e662532f97","Type":"ContainerDied","Data":"65212bb8b26f2669a282d5f81db7c4fce5651a93ef31150e16a23175b9d9c68f"} Sep 16 00:51:24.362814 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:24.362726 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq" event={"ID":"eb5b92ac-00d0-45ca-a614-c1e662532f97","Type":"ContainerDied","Data":"23a0722304b3ee5708e276e408cf9808bd6ca029a9e1023f77e1398da98f189e"} Sep 16 00:51:24.362814 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:24.362755 2567 scope.go:117] "RemoveContainer" containerID="65212bb8b26f2669a282d5f81db7c4fce5651a93ef31150e16a23175b9d9c68f" Sep 16 00:51:24.376585 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:24.376563 2567 scope.go:117] "RemoveContainer" containerID="65212bb8b26f2669a282d5f81db7c4fce5651a93ef31150e16a23175b9d9c68f" Sep 16 00:51:24.377400 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:51:24.377214 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65212bb8b26f2669a282d5f81db7c4fce5651a93ef31150e16a23175b9d9c68f\": container with ID starting with 65212bb8b26f2669a282d5f81db7c4fce5651a93ef31150e16a23175b9d9c68f not found: ID does not exist" containerID="65212bb8b26f2669a282d5f81db7c4fce5651a93ef31150e16a23175b9d9c68f" Sep 16 00:51:24.377400 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:24.377261 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65212bb8b26f2669a282d5f81db7c4fce5651a93ef31150e16a23175b9d9c68f"} err="failed to get container status \"65212bb8b26f2669a282d5f81db7c4fce5651a93ef31150e16a23175b9d9c68f\": rpc error: code = NotFound desc = could not find container \"65212bb8b26f2669a282d5f81db7c4fce5651a93ef31150e16a23175b9d9c68f\": container with ID starting with 65212bb8b26f2669a282d5f81db7c4fce5651a93ef31150e16a23175b9d9c68f not found: ID does not exist" Sep 16 00:51:24.382745 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:24.382661 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq"] Sep 16 00:51:24.387087 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:24.387013 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4d413f/test-model-4d413f-kserve-d8cfcf6d9-zqszq"] Sep 16 00:51:25.292247 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:25.292207 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:25.392450 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:25.392413 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:25.627259 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:25.627217 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:26.120538 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:26.120490 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb5b92ac-00d0-45ca-a614-c1e662532f97" path="/var/lib/kubelet/pods/eb5b92ac-00d0-45ca-a614-c1e662532f97/volumes" Sep 16 00:51:26.329775 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:26.329743 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d413f-rnzcg"] Sep 16 00:51:26.330194 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:26.330172 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb5b92ac-00d0-45ca-a614-c1e662532f97" containerName="main" Sep 16 00:51:26.330194 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:26.330185 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb5b92ac-00d0-45ca-a614-c1e662532f97" containerName="main" Sep 16 00:51:26.330194 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:26.330195 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d35824b-1a22-4c15-b39a-23d9e955d35a" containerName="revoke-keys" Sep 16 00:51:26.330298 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:26.330200 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d35824b-1a22-4c15-b39a-23d9e955d35a" containerName="revoke-keys" Sep 16 00:51:26.330298 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:26.330221 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe6fcd3b-33cf-4706-af7f-54b182eb6821" containerName="payload-processing" Sep 16 00:51:26.330298 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:26.330226 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe6fcd3b-33cf-4706-af7f-54b182eb6821" containerName="payload-processing" Sep 16 00:51:26.330298 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:26.330288 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb5b92ac-00d0-45ca-a614-c1e662532f97" containerName="main" Sep 16 00:51:26.330298 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:26.330296 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d35824b-1a22-4c15-b39a-23d9e955d35a" containerName="revoke-keys" Sep 16 00:51:26.330444 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:26.330306 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe6fcd3b-33cf-4706-af7f-54b182eb6821" containerName="payload-processing" Sep 16 00:51:26.334980 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:26.334957 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d413f-rnzcg" Sep 16 00:51:26.337868 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:26.337841 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 00:51:26.338018 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:26.337892 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-npltg\"" Sep 16 00:51:26.340391 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:26.340355 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d413f-rnzcg"] Sep 16 00:51:26.456404 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:26.456309 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/511db2c0-953d-4df9-a50b-ebbba23bbc32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4d413f-rnzcg\" (UID: \"511db2c0-953d-4df9-a50b-ebbba23bbc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d413f-rnzcg" Sep 16 00:51:26.557853 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:26.557810 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/511db2c0-953d-4df9-a50b-ebbba23bbc32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4d413f-rnzcg\" (UID: \"511db2c0-953d-4df9-a50b-ebbba23bbc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d413f-rnzcg" Sep 16 00:51:26.558568 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:26.558540 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/511db2c0-953d-4df9-a50b-ebbba23bbc32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4d413f-rnzcg\" (UID: \"511db2c0-953d-4df9-a50b-ebbba23bbc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d413f-rnzcg" Sep 16 00:51:26.649395 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:26.649354 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d413f-rnzcg" Sep 16 00:51:26.924472 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:26.924440 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d413f-rnzcg"] Sep 16 00:51:26.925614 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:51:26.925574 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod511db2c0_953d_4df9_a50b_ebbba23bbc32.slice/crio-3e37bd77fd8632f6f74aba7bf763bf66d1eac4bcffb4852c83c9ad4abf83cab7 WatchSource:0}: Error finding container 3e37bd77fd8632f6f74aba7bf763bf66d1eac4bcffb4852c83c9ad4abf83cab7: Status 404 returned error can't find the container with id 3e37bd77fd8632f6f74aba7bf763bf66d1eac4bcffb4852c83c9ad4abf83cab7 Sep 16 00:51:27.386586 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:27.386546 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d413f-rnzcg" event={"ID":"511db2c0-953d-4df9-a50b-ebbba23bbc32","Type":"ContainerStarted","Data":"96ef4f71214b32348289235a0175563280e4b25063c868fdff4a046cb2e554a4"} Sep 16 00:51:27.386984 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:27.386601 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d413f-rnzcg" event={"ID":"511db2c0-953d-4df9-a50b-ebbba23bbc32","Type":"ContainerStarted","Data":"3e37bd77fd8632f6f74aba7bf763bf66d1eac4bcffb4852c83c9ad4abf83cab7"} Sep 16 00:51:27.403300 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:27.403230 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d413f-rnzcg" podStartSLOduration=1.403210383 podStartE2EDuration="1.403210383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:51:27.400876833 +0000 UTC m=+1037.763100391" watchObservedRunningTime="2026-09-16 00:51:27.403210383 +0000 UTC m=+1037.765433915" Sep 16 00:51:27.521495 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:27.521400 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:28.150491 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:28.150457 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:28.394331 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:28.394295 2567 generic.go:356] "Generic (PLEG): container finished" podID="511db2c0-953d-4df9-a50b-ebbba23bbc32" containerID="96ef4f71214b32348289235a0175563280e4b25063c868fdff4a046cb2e554a4" exitCode=0 Sep 16 00:51:28.394708 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:28.394376 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d413f-rnzcg" event={"ID":"511db2c0-953d-4df9-a50b-ebbba23bbc32","Type":"ContainerDied","Data":"96ef4f71214b32348289235a0175563280e4b25063c868fdff4a046cb2e554a4"} Sep 16 00:51:28.817377 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:28.817343 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:29.543608 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:29.543579 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d413f-rnzcg" Sep 16 00:51:29.689512 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:29.689417 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/511db2c0-953d-4df9-a50b-ebbba23bbc32-maas-api-service-ca\") pod \"511db2c0-953d-4df9-a50b-ebbba23bbc32\" (UID: \"511db2c0-953d-4df9-a50b-ebbba23bbc32\") " Sep 16 00:51:29.689793 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:29.689768 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511db2c0-953d-4df9-a50b-ebbba23bbc32-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "511db2c0-953d-4df9-a50b-ebbba23bbc32" (UID: "511db2c0-953d-4df9-a50b-ebbba23bbc32"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:51:29.790328 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:29.790285 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/511db2c0-953d-4df9-a50b-ebbba23bbc32-maas-api-service-ca\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:30.407291 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:30.407200 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d413f-rnzcg" Sep 16 00:51:30.407291 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:30.407247 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d413f-rnzcg" event={"ID":"511db2c0-953d-4df9-a50b-ebbba23bbc32","Type":"ContainerDied","Data":"3e37bd77fd8632f6f74aba7bf763bf66d1eac4bcffb4852c83c9ad4abf83cab7"} Sep 16 00:51:30.407496 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:30.407301 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e37bd77fd8632f6f74aba7bf763bf66d1eac4bcffb4852c83c9ad4abf83cab7" Sep 16 00:51:34.230389 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:34.230355 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:35.976448 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:35.976414 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:36.077413 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:36.077376 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:36.670478 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:36.670445 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:36.864817 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:36.864777 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:36.902361 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:36.902325 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628"] Sep 16 00:51:36.902590 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:36.902564 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628" podUID="b4fd1ad7-62dc-46f0-9069-afe0a5f45e70" containerName="payload-pre-processing" containerID="cri-o://645dcc09cfee83a00248287952ce36a8d93e7c640a5bbdde7d29c7aa6543e342" gracePeriod=30 Sep 16 00:51:36.968730 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:36.968609 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:37.066690 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.066652 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d413f-rnzcg"] Sep 16 00:51:37.067845 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.067820 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:37.069502 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.069473 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d413f-rnzcg"] Sep 16 00:51:37.133684 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.133640 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:37.168691 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.168658 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628" Sep 16 00:51:37.265467 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.265369 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4fd1ad7-62dc-46f0-9069-afe0a5f45e70-plugins-config-volume\") pod \"b4fd1ad7-62dc-46f0-9069-afe0a5f45e70\" (UID: \"b4fd1ad7-62dc-46f0-9069-afe0a5f45e70\") " Sep 16 00:51:37.265646 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.265497 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnwfp\" (UniqueName: \"kubernetes.io/projected/b4fd1ad7-62dc-46f0-9069-afe0a5f45e70-kube-api-access-qnwfp\") pod \"b4fd1ad7-62dc-46f0-9069-afe0a5f45e70\" (UID: \"b4fd1ad7-62dc-46f0-9069-afe0a5f45e70\") " Sep 16 00:51:37.265776 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.265751 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fd1ad7-62dc-46f0-9069-afe0a5f45e70-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b4fd1ad7-62dc-46f0-9069-afe0a5f45e70" (UID: "b4fd1ad7-62dc-46f0-9069-afe0a5f45e70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:51:37.267880 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.267853 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fd1ad7-62dc-46f0-9069-afe0a5f45e70-kube-api-access-qnwfp" (OuterVolumeSpecName: "kube-api-access-qnwfp") pod "b4fd1ad7-62dc-46f0-9069-afe0a5f45e70" (UID: "b4fd1ad7-62dc-46f0-9069-afe0a5f45e70"). InnerVolumeSpecName "kube-api-access-qnwfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:51:37.366447 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.366406 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4fd1ad7-62dc-46f0-9069-afe0a5f45e70-plugins-config-volume\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:37.366447 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.366444 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnwfp\" (UniqueName: \"kubernetes.io/projected/b4fd1ad7-62dc-46f0-9069-afe0a5f45e70-kube-api-access-qnwfp\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:37.457207 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.457170 2567 generic.go:356] "Generic (PLEG): container finished" podID="b4fd1ad7-62dc-46f0-9069-afe0a5f45e70" containerID="645dcc09cfee83a00248287952ce36a8d93e7c640a5bbdde7d29c7aa6543e342" exitCode=0 Sep 16 00:51:37.457398 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.457217 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628" event={"ID":"b4fd1ad7-62dc-46f0-9069-afe0a5f45e70","Type":"ContainerDied","Data":"645dcc09cfee83a00248287952ce36a8d93e7c640a5bbdde7d29c7aa6543e342"} Sep 16 00:51:37.457398 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.457222 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628" Sep 16 00:51:37.457398 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.457260 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628" event={"ID":"b4fd1ad7-62dc-46f0-9069-afe0a5f45e70","Type":"ContainerDied","Data":"e2f5f2b4c3bf137ce2c326ae1c95db65b743b576dfb93fb6160cbe71f6308c20"} Sep 16 00:51:37.457398 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.457283 2567 scope.go:117] "RemoveContainer" containerID="645dcc09cfee83a00248287952ce36a8d93e7c640a5bbdde7d29c7aa6543e342" Sep 16 00:51:37.467855 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.467829 2567 scope.go:117] "RemoveContainer" containerID="645dcc09cfee83a00248287952ce36a8d93e7c640a5bbdde7d29c7aa6543e342" Sep 16 00:51:37.468290 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:51:37.468266 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"645dcc09cfee83a00248287952ce36a8d93e7c640a5bbdde7d29c7aa6543e342\": container with ID starting with 645dcc09cfee83a00248287952ce36a8d93e7c640a5bbdde7d29c7aa6543e342 not found: ID does not exist" containerID="645dcc09cfee83a00248287952ce36a8d93e7c640a5bbdde7d29c7aa6543e342" Sep 16 00:51:37.468393 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.468304 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"645dcc09cfee83a00248287952ce36a8d93e7c640a5bbdde7d29c7aa6543e342"} err="failed to get container status \"645dcc09cfee83a00248287952ce36a8d93e7c640a5bbdde7d29c7aa6543e342\": rpc error: code = NotFound desc = could not find container \"645dcc09cfee83a00248287952ce36a8d93e7c640a5bbdde7d29c7aa6543e342\": container with ID starting with 645dcc09cfee83a00248287952ce36a8d93e7c640a5bbdde7d29c7aa6543e342 not found: ID does not exist" Sep 16 00:51:37.480617 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.480583 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628"] Sep 16 00:51:37.488504 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.488445 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4d413f-6dc5fc6d94-74628"] Sep 16 00:51:37.518172 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.518025 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:37.619826 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:37.619795 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:38.121497 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:38.121448 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="511db2c0-953d-4df9-a50b-ebbba23bbc32" path="/var/lib/kubelet/pods/511db2c0-953d-4df9-a50b-ebbba23bbc32/volumes" Sep 16 00:51:38.122109 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:38.122091 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4fd1ad7-62dc-46f0-9069-afe0a5f45e70" path="/var/lib/kubelet/pods/b4fd1ad7-62dc-46f0-9069-afe0a5f45e70/volumes" Sep 16 00:51:38.327438 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:38.327391 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:42.313882 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:42.313851 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:42.414643 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:42.414605 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:42.552214 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:42.552180 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:42.863744 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:42.863685 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:43.028313 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:43.028271 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:43.605496 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:43.601912 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:43.662518 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:43.662482 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:43.859303 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:43.859211 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l"] Sep 16 00:51:43.859506 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:43.859478 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" podUID="43f48e12-f207-480b-99c3-51eea6a1a852" containerName="istio-proxy" containerID="cri-o://4d3303125015a673f89b8b7306d11ceb9d4d030f69ca73615be3b85d5b5ad467" gracePeriod=30 Sep 16 00:51:43.873659 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:43.873626 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:44.473694 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:44.473659 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:44.761189 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:44.761081 2567 patch_prober.go:28] interesting pod/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 00:51:44.761189 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:44.761158 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" podUID="43f48e12-f207-480b-99c3-51eea6a1a852" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 00:51:44.808655 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:44.808600 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:45.026438 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:45.026343 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:45.594731 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:45.594695 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:45.698445 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:45.698413 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:46.642998 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:46.642946 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:47.539826 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:47.539794 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:47.638973 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:47.638936 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:47.755503 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:47.755471 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:48.171704 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.171670 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:48.266351 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.266319 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:48.367336 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.367285 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:48.388169 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.388135 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh"] Sep 16 00:51:48.388569 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.388544 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4fd1ad7-62dc-46f0-9069-afe0a5f45e70" containerName="payload-pre-processing" Sep 16 00:51:48.388569 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.388565 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4fd1ad7-62dc-46f0-9069-afe0a5f45e70" containerName="payload-pre-processing" Sep 16 00:51:48.388772 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.388582 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="511db2c0-953d-4df9-a50b-ebbba23bbc32" containerName="revoke-keys" Sep 16 00:51:48.388772 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.388590 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="511db2c0-953d-4df9-a50b-ebbba23bbc32" containerName="revoke-keys" Sep 16 00:51:48.388772 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.388678 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="511db2c0-953d-4df9-a50b-ebbba23bbc32" containerName="revoke-keys" Sep 16 00:51:48.388772 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.388695 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4fd1ad7-62dc-46f0-9069-afe0a5f45e70" containerName="payload-pre-processing" Sep 16 00:51:48.393100 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.393074 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh" Sep 16 00:51:48.395726 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.395701 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-629e17ec-dockercfg-9xjkk\"" Sep 16 00:51:48.395866 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.395748 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-629e17ec\"" Sep 16 00:51:48.400449 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.400423 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh"] Sep 16 00:51:48.480897 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.480811 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z"] Sep 16 00:51:48.484409 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.484385 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z" Sep 16 00:51:48.486294 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.486266 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gtt4\" (UniqueName: \"kubernetes.io/projected/7ef9fff3-75e1-4e17-a195-19c513aee1cc-kube-api-access-7gtt4\") pod \"payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh\" (UID: \"7ef9fff3-75e1-4e17-a195-19c513aee1cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh" Sep 16 00:51:48.486426 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.486367 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef9fff3-75e1-4e17-a195-19c513aee1cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh\" (UID: \"7ef9fff3-75e1-4e17-a195-19c513aee1cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh" Sep 16 00:51:48.495688 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.495656 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z"] Sep 16 00:51:48.588235 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.588185 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqhbg\" (UniqueName: \"kubernetes.io/projected/d073679f-261b-42a6-9584-42927af437c5-kube-api-access-cqhbg\") pod \"payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z\" (UID: \"d073679f-261b-42a6-9584-42927af437c5\") " pod="openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z" Sep 16 00:51:48.588431 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.588325 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gtt4\" (UniqueName: \"kubernetes.io/projected/7ef9fff3-75e1-4e17-a195-19c513aee1cc-kube-api-access-7gtt4\") pod \"payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh\" (UID: \"7ef9fff3-75e1-4e17-a195-19c513aee1cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh" Sep 16 00:51:48.588431 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.588383 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d073679f-261b-42a6-9584-42927af437c5-plugins-config-volume\") pod \"payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z\" (UID: \"d073679f-261b-42a6-9584-42927af437c5\") " pod="openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z" Sep 16 00:51:48.588431 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.588417 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef9fff3-75e1-4e17-a195-19c513aee1cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh\" (UID: \"7ef9fff3-75e1-4e17-a195-19c513aee1cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh" Sep 16 00:51:48.589189 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.589166 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef9fff3-75e1-4e17-a195-19c513aee1cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh\" (UID: \"7ef9fff3-75e1-4e17-a195-19c513aee1cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh" Sep 16 00:51:48.597489 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.597459 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gtt4\" (UniqueName: \"kubernetes.io/projected/7ef9fff3-75e1-4e17-a195-19c513aee1cc-kube-api-access-7gtt4\") pod \"payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh\" (UID: \"7ef9fff3-75e1-4e17-a195-19c513aee1cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh" Sep 16 00:51:48.689741 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.689703 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqhbg\" (UniqueName: \"kubernetes.io/projected/d073679f-261b-42a6-9584-42927af437c5-kube-api-access-cqhbg\") pod \"payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z\" (UID: \"d073679f-261b-42a6-9584-42927af437c5\") " pod="openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z" Sep 16 00:51:48.689932 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.689784 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d073679f-261b-42a6-9584-42927af437c5-plugins-config-volume\") pod \"payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z\" (UID: \"d073679f-261b-42a6-9584-42927af437c5\") " pod="openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z" Sep 16 00:51:48.690477 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.690457 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d073679f-261b-42a6-9584-42927af437c5-plugins-config-volume\") pod \"payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z\" (UID: \"d073679f-261b-42a6-9584-42927af437c5\") " pod="openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z" Sep 16 00:51:48.697939 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.697910 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqhbg\" (UniqueName: \"kubernetes.io/projected/d073679f-261b-42a6-9584-42927af437c5-kube-api-access-cqhbg\") pod \"payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z\" (UID: \"d073679f-261b-42a6-9584-42927af437c5\") " pod="openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z" Sep 16 00:51:48.705755 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.705727 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh" Sep 16 00:51:48.796279 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.796247 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z" Sep 16 00:51:48.913094 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.913052 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:48.984162 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:48.984128 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh"] Sep 16 00:51:48.984589 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:51:48.984564 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef9fff3_75e1_4e17_a195_19c513aee1cc.slice/crio-44399b4031aadff455851b5279eb1e799058eb8710f9cb4becf5d2f5a9b95342 WatchSource:0}: Error finding container 44399b4031aadff455851b5279eb1e799058eb8710f9cb4becf5d2f5a9b95342: Status 404 returned error can't find the container with id 44399b4031aadff455851b5279eb1e799058eb8710f9cb4becf5d2f5a9b95342 Sep 16 00:51:49.076524 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.076466 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z"] Sep 16 00:51:49.077567 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:51:49.077532 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd073679f_261b_42a6_9584_42927af437c5.slice/crio-b4ab395092309240a1e48dc6709805cea11df84a91c31413c414a9e0962eb80f WatchSource:0}: Error finding container b4ab395092309240a1e48dc6709805cea11df84a91c31413c414a9e0962eb80f: Status 404 returned error can't find the container with id b4ab395092309240a1e48dc6709805cea11df84a91c31413c414a9e0962eb80f Sep 16 00:51:49.133304 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.133271 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:51:49.192865 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.192828 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43f48e12-f207-480b-99c3-51eea6a1a852-istio-token\") pod \"43f48e12-f207-480b-99c3-51eea6a1a852\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " Sep 16 00:51:49.193170 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.192895 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-workload-certs\") pod \"43f48e12-f207-480b-99c3-51eea6a1a852\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " Sep 16 00:51:49.193170 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.192941 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43f48e12-f207-480b-99c3-51eea6a1a852-istio-podinfo\") pod \"43f48e12-f207-480b-99c3-51eea6a1a852\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " Sep 16 00:51:49.193170 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.192977 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-istio-envoy\") pod \"43f48e12-f207-480b-99c3-51eea6a1a852\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " Sep 16 00:51:49.193170 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.193013 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6lxl\" (UniqueName: \"kubernetes.io/projected/43f48e12-f207-480b-99c3-51eea6a1a852-kube-api-access-v6lxl\") pod \"43f48e12-f207-480b-99c3-51eea6a1a852\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " Sep 16 00:51:49.193170 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.193069 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-credential-socket\") pod \"43f48e12-f207-480b-99c3-51eea6a1a852\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " Sep 16 00:51:49.193452 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.193184 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-workload-socket\") pod \"43f48e12-f207-480b-99c3-51eea6a1a852\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " Sep 16 00:51:49.193452 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.193215 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43f48e12-f207-480b-99c3-51eea6a1a852-istiod-ca-cert\") pod \"43f48e12-f207-480b-99c3-51eea6a1a852\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " Sep 16 00:51:49.193452 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.193243 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-istio-data\") pod \"43f48e12-f207-480b-99c3-51eea6a1a852\" (UID: \"43f48e12-f207-480b-99c3-51eea6a1a852\") " Sep 16 00:51:49.193452 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.193307 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "43f48e12-f207-480b-99c3-51eea6a1a852" (UID: "43f48e12-f207-480b-99c3-51eea6a1a852"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:51:49.193656 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.193512 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "43f48e12-f207-480b-99c3-51eea6a1a852" (UID: "43f48e12-f207-480b-99c3-51eea6a1a852"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:51:49.193656 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.193525 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-istio-data" (OuterVolumeSpecName: "istio-data") pod "43f48e12-f207-480b-99c3-51eea6a1a852" (UID: "43f48e12-f207-480b-99c3-51eea6a1a852"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:51:49.193765 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.193705 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-workload-certs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:49.193765 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.193724 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-workload-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:49.193765 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.193694 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "43f48e12-f207-480b-99c3-51eea6a1a852" (UID: "43f48e12-f207-480b-99c3-51eea6a1a852"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:51:49.193765 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.193738 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-istio-data\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:49.194021 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.193999 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f48e12-f207-480b-99c3-51eea6a1a852-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "43f48e12-f207-480b-99c3-51eea6a1a852" (UID: "43f48e12-f207-480b-99c3-51eea6a1a852"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:51:49.195844 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.195814 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f48e12-f207-480b-99c3-51eea6a1a852-kube-api-access-v6lxl" (OuterVolumeSpecName: "kube-api-access-v6lxl") pod "43f48e12-f207-480b-99c3-51eea6a1a852" (UID: "43f48e12-f207-480b-99c3-51eea6a1a852"). InnerVolumeSpecName "kube-api-access-v6lxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:51:49.196132 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.196108 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43f48e12-f207-480b-99c3-51eea6a1a852-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "43f48e12-f207-480b-99c3-51eea6a1a852" (UID: "43f48e12-f207-480b-99c3-51eea6a1a852"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 00:51:49.196189 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.196141 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "43f48e12-f207-480b-99c3-51eea6a1a852" (UID: "43f48e12-f207-480b-99c3-51eea6a1a852"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:51:49.196189 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.196177 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f48e12-f207-480b-99c3-51eea6a1a852-istio-token" (OuterVolumeSpecName: "istio-token") pod "43f48e12-f207-480b-99c3-51eea6a1a852" (UID: "43f48e12-f207-480b-99c3-51eea6a1a852"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:51:49.294992 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.294894 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43f48e12-f207-480b-99c3-51eea6a1a852-istiod-ca-cert\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:49.294992 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.294931 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43f48e12-f207-480b-99c3-51eea6a1a852-istio-token\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:49.294992 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.294945 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43f48e12-f207-480b-99c3-51eea6a1a852-istio-podinfo\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:49.294992 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.294958 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-istio-envoy\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:49.294992 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.294971 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6lxl\" (UniqueName: \"kubernetes.io/projected/43f48e12-f207-480b-99c3-51eea6a1a852-kube-api-access-v6lxl\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:49.294992 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.294985 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f48e12-f207-480b-99c3-51eea6a1a852-credential-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:51:49.532227 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.532187 2567 generic.go:356] "Generic (PLEG): container finished" podID="43f48e12-f207-480b-99c3-51eea6a1a852" containerID="4d3303125015a673f89b8b7306d11ceb9d4d030f69ca73615be3b85d5b5ad467" exitCode=0 Sep 16 00:51:49.532406 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.532239 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" Sep 16 00:51:49.532406 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.532275 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" event={"ID":"43f48e12-f207-480b-99c3-51eea6a1a852","Type":"ContainerDied","Data":"4d3303125015a673f89b8b7306d11ceb9d4d030f69ca73615be3b85d5b5ad467"} Sep 16 00:51:49.532406 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.532338 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l" event={"ID":"43f48e12-f207-480b-99c3-51eea6a1a852","Type":"ContainerDied","Data":"5dcdddbf19bc98bfc10ca4c9bd50b55aadfb483a324e53de0a7ceda2828a7462"} Sep 16 00:51:49.532406 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.532371 2567 scope.go:117] "RemoveContainer" containerID="4d3303125015a673f89b8b7306d11ceb9d4d030f69ca73615be3b85d5b5ad467" Sep 16 00:51:49.534307 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.534278 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh" event={"ID":"7ef9fff3-75e1-4e17-a195-19c513aee1cc","Type":"ContainerStarted","Data":"3d9b25fdc72504276b6be424048bfdcff4481312e10e94a20a730af9ce5a039d"} Sep 16 00:51:49.534452 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.534340 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh" event={"ID":"7ef9fff3-75e1-4e17-a195-19c513aee1cc","Type":"ContainerStarted","Data":"44399b4031aadff455851b5279eb1e799058eb8710f9cb4becf5d2f5a9b95342"} Sep 16 00:51:49.534452 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.534371 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh" Sep 16 00:51:49.536127 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.536099 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z" event={"ID":"d073679f-261b-42a6-9584-42927af437c5","Type":"ContainerStarted","Data":"e04ab29d012017eb5abe11e0a917c55f6ae733685de4c32ece678af16188f7f1"} Sep 16 00:51:49.536263 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.536153 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z" event={"ID":"d073679f-261b-42a6-9584-42927af437c5","Type":"ContainerStarted","Data":"b4ab395092309240a1e48dc6709805cea11df84a91c31413c414a9e0962eb80f"} Sep 16 00:51:49.536263 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.536189 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z" Sep 16 00:51:49.545433 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.545408 2567 scope.go:117] "RemoveContainer" containerID="2efa8462e03e07e86534304fde60276d34e90fcae0f1d50476d73e6bfa4f91b7" Sep 16 00:51:49.551506 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.551452 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh" podStartSLOduration=1.551433931 podStartE2EDuration="1.551433931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:51:49.550332454 +0000 UTC m=+1059.912556010" watchObservedRunningTime="2026-09-16 00:51:49.551433931 +0000 UTC m=+1059.913657466" Sep 16 00:51:49.557798 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.557763 2567 scope.go:117] "RemoveContainer" containerID="4d3303125015a673f89b8b7306d11ceb9d4d030f69ca73615be3b85d5b5ad467" Sep 16 00:51:49.558409 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:51:49.558375 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d3303125015a673f89b8b7306d11ceb9d4d030f69ca73615be3b85d5b5ad467\": container with ID starting with 4d3303125015a673f89b8b7306d11ceb9d4d030f69ca73615be3b85d5b5ad467 not found: ID does not exist" containerID="4d3303125015a673f89b8b7306d11ceb9d4d030f69ca73615be3b85d5b5ad467" Sep 16 00:51:49.558515 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.558422 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d3303125015a673f89b8b7306d11ceb9d4d030f69ca73615be3b85d5b5ad467"} err="failed to get container status \"4d3303125015a673f89b8b7306d11ceb9d4d030f69ca73615be3b85d5b5ad467\": rpc error: code = NotFound desc = could not find container \"4d3303125015a673f89b8b7306d11ceb9d4d030f69ca73615be3b85d5b5ad467\": container with ID starting with 4d3303125015a673f89b8b7306d11ceb9d4d030f69ca73615be3b85d5b5ad467 not found: ID does not exist" Sep 16 00:51:49.558515 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.558448 2567 scope.go:117] "RemoveContainer" containerID="2efa8462e03e07e86534304fde60276d34e90fcae0f1d50476d73e6bfa4f91b7" Sep 16 00:51:49.558801 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:51:49.558782 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2efa8462e03e07e86534304fde60276d34e90fcae0f1d50476d73e6bfa4f91b7\": container with ID starting with 2efa8462e03e07e86534304fde60276d34e90fcae0f1d50476d73e6bfa4f91b7 not found: ID does not exist" containerID="2efa8462e03e07e86534304fde60276d34e90fcae0f1d50476d73e6bfa4f91b7" Sep 16 00:51:49.558850 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.558808 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2efa8462e03e07e86534304fde60276d34e90fcae0f1d50476d73e6bfa4f91b7"} err="failed to get container status \"2efa8462e03e07e86534304fde60276d34e90fcae0f1d50476d73e6bfa4f91b7\": rpc error: code = NotFound desc = could not find container \"2efa8462e03e07e86534304fde60276d34e90fcae0f1d50476d73e6bfa4f91b7\": container with ID starting with 2efa8462e03e07e86534304fde60276d34e90fcae0f1d50476d73e6bfa4f91b7 not found: ID does not exist" Sep 16 00:51:49.568271 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.568215 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z" podStartSLOduration=1.568196526 podStartE2EDuration="1.568196526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:51:49.565292864 +0000 UTC m=+1059.927516416" watchObservedRunningTime="2026-09-16 00:51:49.568196526 +0000 UTC m=+1059.930420058" Sep 16 00:51:49.583751 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.583682 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l"] Sep 16 00:51:49.587588 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.587560 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4d413f-openshift-default-65896f7dd8-xcw9l"] Sep 16 00:51:49.922820 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:49.922784 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:50.039488 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:50.039451 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:51:50.119322 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:50.119288 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43f48e12-f207-480b-99c3-51eea6a1a852" path="/var/lib/kubelet/pods/43f48e12-f207-480b-99c3-51eea6a1a852/volumes" Sep 16 00:51:50.629391 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:50.629351 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:51.332398 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:51.332359 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:51.430157 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:51.430125 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:51.709238 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:51.709148 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:51.810903 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:51.810862 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:51.897659 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:51.897623 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:51:51.934298 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:51.934263 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v"] Sep 16 00:51:51.934668 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:51.934656 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f48e12-f207-480b-99c3-51eea6a1a852" containerName="istio-proxy" Sep 16 00:51:51.934720 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:51.934670 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f48e12-f207-480b-99c3-51eea6a1a852" containerName="istio-proxy" Sep 16 00:51:51.934759 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:51.934744 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="43f48e12-f207-480b-99c3-51eea6a1a852" containerName="istio-proxy" Sep 16 00:51:51.934759 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:51.934752 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="43f48e12-f207-480b-99c3-51eea6a1a852" containerName="istio-proxy" Sep 16 00:51:51.939630 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:51.939595 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v" Sep 16 00:51:51.942385 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:51.942357 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-108960\"" Sep 16 00:51:51.942628 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:51.942469 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-108960-dockercfg-qxkxd\"" Sep 16 00:51:51.946829 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:51.946802 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v"] Sep 16 00:51:52.022490 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:52.022367 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsdlr\" (UniqueName: \"kubernetes.io/projected/08dd2134-017e-48bf-a2ac-96ee9984931e-kube-api-access-zsdlr\") pod \"payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v\" (UID: \"08dd2134-017e-48bf-a2ac-96ee9984931e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v" Sep 16 00:51:52.022490 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:52.022441 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08dd2134-017e-48bf-a2ac-96ee9984931e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v\" (UID: \"08dd2134-017e-48bf-a2ac-96ee9984931e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v" Sep 16 00:51:52.123510 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:52.123468 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsdlr\" (UniqueName: \"kubernetes.io/projected/08dd2134-017e-48bf-a2ac-96ee9984931e-kube-api-access-zsdlr\") pod \"payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v\" (UID: \"08dd2134-017e-48bf-a2ac-96ee9984931e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v" Sep 16 00:51:52.123717 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:52.123519 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08dd2134-017e-48bf-a2ac-96ee9984931e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v\" (UID: \"08dd2134-017e-48bf-a2ac-96ee9984931e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v" Sep 16 00:51:52.124258 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:52.124235 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08dd2134-017e-48bf-a2ac-96ee9984931e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v\" (UID: \"08dd2134-017e-48bf-a2ac-96ee9984931e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v" Sep 16 00:51:52.131906 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:52.131870 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsdlr\" (UniqueName: \"kubernetes.io/projected/08dd2134-017e-48bf-a2ac-96ee9984931e-kube-api-access-zsdlr\") pod \"payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v\" (UID: \"08dd2134-017e-48bf-a2ac-96ee9984931e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v" Sep 16 00:51:52.252325 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:52.252288 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v" Sep 16 00:51:52.538806 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:52.538768 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v"] Sep 16 00:51:52.540117 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:51:52.540078 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08dd2134_017e_48bf_a2ac_96ee9984931e.slice/crio-4d70f89a7b270bebf3d1e4a2c365253129806829b1643184528811af6d9968af WatchSource:0}: Error finding container 4d70f89a7b270bebf3d1e4a2c365253129806829b1643184528811af6d9968af: Status 404 returned error can't find the container with id 4d70f89a7b270bebf3d1e4a2c365253129806829b1643184528811af6d9968af Sep 16 00:51:52.559481 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:52.559394 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v" event={"ID":"08dd2134-017e-48bf-a2ac-96ee9984931e","Type":"ContainerStarted","Data":"4d70f89a7b270bebf3d1e4a2c365253129806829b1643184528811af6d9968af"} Sep 16 00:51:53.570007 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:53.569967 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v" event={"ID":"08dd2134-017e-48bf-a2ac-96ee9984931e","Type":"ContainerStarted","Data":"8fb868e44aadd38d9eb2beedd33ff7da8a5eb29170cf07cc64915e20e8307a0c"} Sep 16 00:51:53.570531 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:53.570056 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v" Sep 16 00:51:53.586827 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:51:53.586773 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v" podStartSLOduration=2.586753613 podStartE2EDuration="2.586753613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:51:53.585076306 +0000 UTC m=+1063.947299861" watchObservedRunningTime="2026-09-16 00:51:53.586753613 +0000 UTC m=+1063.948977145" Sep 16 00:52:00.545569 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:00.545532 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z" Sep 16 00:52:00.546053 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:00.545653 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh" Sep 16 00:52:04.577829 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:04.577745 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v" Sep 16 00:52:08.862527 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:08.862489 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:10.610823 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:10.610780 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:11.102651 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:11.102609 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-629e17ec-lwgrf"] Sep 16 00:52:11.103029 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:11.103015 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f48e12-f207-480b-99c3-51eea6a1a852" containerName="istio-proxy" Sep 16 00:52:11.103126 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:11.103052 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f48e12-f207-480b-99c3-51eea6a1a852" containerName="istio-proxy" Sep 16 00:52:11.107776 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:11.107752 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-629e17ec-lwgrf" Sep 16 00:52:11.110683 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:11.110656 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 00:52:11.110958 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:11.110664 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-npltg\"" Sep 16 00:52:11.113228 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:11.113201 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-629e17ec-lwgrf"] Sep 16 00:52:11.219367 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:11.219325 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-629e17ec-lwgrf\" (UID: \"eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-629e17ec-lwgrf" Sep 16 00:52:11.320126 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:11.320084 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-629e17ec-lwgrf\" (UID: \"eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-629e17ec-lwgrf" Sep 16 00:52:11.320718 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:11.320696 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-629e17ec-lwgrf\" (UID: \"eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-629e17ec-lwgrf" Sep 16 00:52:11.420886 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:11.420796 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-629e17ec-lwgrf" Sep 16 00:52:11.671838 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:11.671748 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:11.698072 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:11.698026 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-629e17ec-lwgrf"] Sep 16 00:52:11.699325 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:52:11.699295 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef5ea07_2db4_45b0_b4f8_a9d4abfc9cb6.slice/crio-25f1cba5776c76c12d19a3c6196515f53b8c9787d2964c7969bc29d1bc7b79e9 WatchSource:0}: Error finding container 25f1cba5776c76c12d19a3c6196515f53b8c9787d2964c7969bc29d1bc7b79e9: Status 404 returned error can't find the container with id 25f1cba5776c76c12d19a3c6196515f53b8c9787d2964c7969bc29d1bc7b79e9 Sep 16 00:52:12.687998 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:12.687891 2567 generic.go:356] "Generic (PLEG): container finished" podID="eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6" containerID="bd461d9b584f5c983822b52c525d696369d98d03a7b50caf893e5f9209348bfb" exitCode=0 Sep 16 00:52:12.688418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:12.687997 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-629e17ec-lwgrf" event={"ID":"eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6","Type":"ContainerDied","Data":"bd461d9b584f5c983822b52c525d696369d98d03a7b50caf893e5f9209348bfb"} Sep 16 00:52:12.688418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:12.688077 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-629e17ec-lwgrf" event={"ID":"eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6","Type":"ContainerStarted","Data":"25f1cba5776c76c12d19a3c6196515f53b8c9787d2964c7969bc29d1bc7b79e9"} Sep 16 00:52:13.017332 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.017237 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:13.143946 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.143891 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:13.493907 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.493856 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:13.798394 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.798346 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8"] Sep 16 00:52:13.802734 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.802703 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:13.805829 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.805796 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-2fa86d-openshift-default-dockercfg-7qwch\"" Sep 16 00:52:13.816860 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.816805 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8"] Sep 16 00:52:13.818056 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.817995 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:13.852099 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.852072 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-629e17ec-lwgrf" Sep 16 00:52:13.883146 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.883114 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:13.947928 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.947883 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6-maas-api-service-ca\") pod \"eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6\" (UID: \"eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6\") " Sep 16 00:52:13.948173 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.948094 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f89837a-e1cf-4d80-a25c-145602b37ca7-istiod-ca-cert\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:13.948250 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.948173 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-token\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:13.948250 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.948203 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-workload-socket\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:13.948250 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.948221 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-workload-certs\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:13.948250 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.948246 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-envoy\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:13.948461 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.948352 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-credential-socket\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:13.948461 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.948383 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6" (UID: "eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:52:13.948461 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.948411 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-podinfo\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:13.948616 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.948504 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-data\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:13.948616 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.948541 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rspj\" (UniqueName: \"kubernetes.io/projected/0f89837a-e1cf-4d80-a25c-145602b37ca7-kube-api-access-5rspj\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:13.948616 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:13.948585 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6-maas-api-service-ca\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:52:14.050122 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.050088 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-token\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:14.050311 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.050133 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-workload-socket\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:14.050311 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.050151 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-workload-certs\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:14.050311 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.050168 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-envoy\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:14.050311 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.050231 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-credential-socket\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:14.050311 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.050270 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-podinfo\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:14.050557 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.050333 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-data\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:14.050557 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.050367 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rspj\" (UniqueName: \"kubernetes.io/projected/0f89837a-e1cf-4d80-a25c-145602b37ca7-kube-api-access-5rspj\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:14.050557 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.050399 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f89837a-e1cf-4d80-a25c-145602b37ca7-istiod-ca-cert\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:14.050717 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.050634 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-workload-certs\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:14.050717 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.050662 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-workload-socket\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:14.050821 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.050764 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-credential-socket\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:14.050967 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.050946 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-data\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:14.051101 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.051080 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f89837a-e1cf-4d80-a25c-145602b37ca7-istiod-ca-cert\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:14.052948 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.052924 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-envoy\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:14.053210 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.053190 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-podinfo\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:14.058177 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.058149 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-token\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:14.058290 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.058204 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rspj\" (UniqueName: \"kubernetes.io/projected/0f89837a-e1cf-4d80-a25c-145602b37ca7-kube-api-access-5rspj\") pod \"e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:14.119725 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.119681 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:14.282082 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.282031 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:14.404171 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.404142 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8"] Sep 16 00:52:14.406555 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:52:14.406513 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f89837a_e1cf_4d80_a25c_145602b37ca7.slice/crio-4d51d99f9b12cb19c03fe27f788e3fc4b8af0b141a5a94de2657a1f2edaac0dd WatchSource:0}: Error finding container 4d51d99f9b12cb19c03fe27f788e3fc4b8af0b141a5a94de2657a1f2edaac0dd: Status 404 returned error can't find the container with id 4d51d99f9b12cb19c03fe27f788e3fc4b8af0b141a5a94de2657a1f2edaac0dd Sep 16 00:52:14.408925 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.408893 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:52:14.409004 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.408968 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:52:14.409073 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.409013 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:52:14.645872 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.645837 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:14.702794 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.702683 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" event={"ID":"0f89837a-e1cf-4d80-a25c-145602b37ca7","Type":"ContainerStarted","Data":"18e6ebca4deef2ef900e844b5909b463b99da48dcd0879cf816874aedd447665"} Sep 16 00:52:14.702794 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.702759 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" event={"ID":"0f89837a-e1cf-4d80-a25c-145602b37ca7","Type":"ContainerStarted","Data":"4d51d99f9b12cb19c03fe27f788e3fc4b8af0b141a5a94de2657a1f2edaac0dd"} Sep 16 00:52:14.704355 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.704328 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-629e17ec-lwgrf" event={"ID":"eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6","Type":"ContainerDied","Data":"25f1cba5776c76c12d19a3c6196515f53b8c9787d2964c7969bc29d1bc7b79e9"} Sep 16 00:52:14.704508 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.704371 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25f1cba5776c76c12d19a3c6196515f53b8c9787d2964c7969bc29d1bc7b79e9" Sep 16 00:52:14.704508 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.704368 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-629e17ec-lwgrf" Sep 16 00:52:14.723367 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:14.723301 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" podStartSLOduration=1.723279365 podStartE2EDuration="1.723279365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:52:14.722744948 +0000 UTC m=+1085.084968526" watchObservedRunningTime="2026-09-16 00:52:14.723279365 +0000 UTC m=+1085.085502898" Sep 16 00:52:15.120416 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:15.120374 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:15.126607 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:15.126572 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:15.583593 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:15.583558 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:15.711121 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:15.711090 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:15.712296 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:15.712269 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:16.680648 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:16.680592 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:16.795406 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:16.795368 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:16.887498 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:16.887448 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:17.647211 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:17.647168 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:17.673026 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:17.672969 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z"] Sep 16 00:52:17.673318 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:17.673285 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z" podUID="d073679f-261b-42a6-9584-42927af437c5" containerName="payload-processing" containerID="cri-o://e04ab29d012017eb5abe11e0a917c55f6ae733685de4c32ece678af16188f7f1" gracePeriod=30 Sep 16 00:52:17.687688 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:17.687649 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh"] Sep 16 00:52:17.688215 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:17.687903 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh" podUID="7ef9fff3-75e1-4e17-a195-19c513aee1cc" containerName="payload-pre-processing" containerID="cri-o://3d9b25fdc72504276b6be424048bfdcff4481312e10e94a20a730af9ce5a039d" gracePeriod=30 Sep 16 00:52:17.730359 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:17.730321 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:17.788434 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:17.788401 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:17.850121 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:17.849935 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:52:17.889642 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:17.889429 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:17.949103 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:17.949069 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:17.950708 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:17.950681 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-629e17ec-lwgrf"] Sep 16 00:52:17.954485 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:17.954454 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-629e17ec-lwgrf"] Sep 16 00:52:17.963512 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:17.963484 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z" Sep 16 00:52:18.002866 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.002831 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh" Sep 16 00:52:18.085240 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.085203 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:18.100387 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.100351 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef9fff3-75e1-4e17-a195-19c513aee1cc-plugins-config-volume\") pod \"7ef9fff3-75e1-4e17-a195-19c513aee1cc\" (UID: \"7ef9fff3-75e1-4e17-a195-19c513aee1cc\") " Sep 16 00:52:18.100566 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.100434 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gtt4\" (UniqueName: \"kubernetes.io/projected/7ef9fff3-75e1-4e17-a195-19c513aee1cc-kube-api-access-7gtt4\") pod \"7ef9fff3-75e1-4e17-a195-19c513aee1cc\" (UID: \"7ef9fff3-75e1-4e17-a195-19c513aee1cc\") " Sep 16 00:52:18.100566 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.100483 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqhbg\" (UniqueName: \"kubernetes.io/projected/d073679f-261b-42a6-9584-42927af437c5-kube-api-access-cqhbg\") pod \"d073679f-261b-42a6-9584-42927af437c5\" (UID: \"d073679f-261b-42a6-9584-42927af437c5\") " Sep 16 00:52:18.100566 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.100503 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d073679f-261b-42a6-9584-42927af437c5-plugins-config-volume\") pod \"d073679f-261b-42a6-9584-42927af437c5\" (UID: \"d073679f-261b-42a6-9584-42927af437c5\") " Sep 16 00:52:18.100889 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.100857 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ef9fff3-75e1-4e17-a195-19c513aee1cc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ef9fff3-75e1-4e17-a195-19c513aee1cc" (UID: "7ef9fff3-75e1-4e17-a195-19c513aee1cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:52:18.101072 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.100917 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d073679f-261b-42a6-9584-42927af437c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d073679f-261b-42a6-9584-42927af437c5" (UID: "d073679f-261b-42a6-9584-42927af437c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:52:18.102826 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.102797 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef9fff3-75e1-4e17-a195-19c513aee1cc-kube-api-access-7gtt4" (OuterVolumeSpecName: "kube-api-access-7gtt4") pod "7ef9fff3-75e1-4e17-a195-19c513aee1cc" (UID: "7ef9fff3-75e1-4e17-a195-19c513aee1cc"). InnerVolumeSpecName "kube-api-access-7gtt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:52:18.102944 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.102884 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d073679f-261b-42a6-9584-42927af437c5-kube-api-access-cqhbg" (OuterVolumeSpecName: "kube-api-access-cqhbg") pod "d073679f-261b-42a6-9584-42927af437c5" (UID: "d073679f-261b-42a6-9584-42927af437c5"). InnerVolumeSpecName "kube-api-access-cqhbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:52:18.119762 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.119723 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6" path="/var/lib/kubelet/pods/eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6/volumes" Sep 16 00:52:18.182101 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.182013 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:18.201337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.201303 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gtt4\" (UniqueName: \"kubernetes.io/projected/7ef9fff3-75e1-4e17-a195-19c513aee1cc-kube-api-access-7gtt4\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:52:18.201337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.201336 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqhbg\" (UniqueName: \"kubernetes.io/projected/d073679f-261b-42a6-9584-42927af437c5-kube-api-access-cqhbg\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:52:18.201566 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.201349 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d073679f-261b-42a6-9584-42927af437c5-plugins-config-volume\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:52:18.201566 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.201362 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef9fff3-75e1-4e17-a195-19c513aee1cc-plugins-config-volume\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:52:18.622466 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.622422 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:18.712847 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.712809 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:18.733642 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.733608 2567 generic.go:356] "Generic (PLEG): container finished" podID="d073679f-261b-42a6-9584-42927af437c5" containerID="e04ab29d012017eb5abe11e0a917c55f6ae733685de4c32ece678af16188f7f1" exitCode=0 Sep 16 00:52:18.733825 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.733671 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z" Sep 16 00:52:18.733825 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.733693 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z" event={"ID":"d073679f-261b-42a6-9584-42927af437c5","Type":"ContainerDied","Data":"e04ab29d012017eb5abe11e0a917c55f6ae733685de4c32ece678af16188f7f1"} Sep 16 00:52:18.733825 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.733748 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z" event={"ID":"d073679f-261b-42a6-9584-42927af437c5","Type":"ContainerDied","Data":"b4ab395092309240a1e48dc6709805cea11df84a91c31413c414a9e0962eb80f"} Sep 16 00:52:18.733825 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.733777 2567 scope.go:117] "RemoveContainer" containerID="e04ab29d012017eb5abe11e0a917c55f6ae733685de4c32ece678af16188f7f1" Sep 16 00:52:18.735299 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.735240 2567 generic.go:356] "Generic (PLEG): container finished" podID="7ef9fff3-75e1-4e17-a195-19c513aee1cc" containerID="3d9b25fdc72504276b6be424048bfdcff4481312e10e94a20a730af9ce5a039d" exitCode=0 Sep 16 00:52:18.735377 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.735319 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh" Sep 16 00:52:18.735434 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.735323 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh" event={"ID":"7ef9fff3-75e1-4e17-a195-19c513aee1cc","Type":"ContainerDied","Data":"3d9b25fdc72504276b6be424048bfdcff4481312e10e94a20a730af9ce5a039d"} Sep 16 00:52:18.735434 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.735427 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh" event={"ID":"7ef9fff3-75e1-4e17-a195-19c513aee1cc","Type":"ContainerDied","Data":"44399b4031aadff455851b5279eb1e799058eb8710f9cb4becf5d2f5a9b95342"} Sep 16 00:52:18.744641 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.744618 2567 scope.go:117] "RemoveContainer" containerID="e04ab29d012017eb5abe11e0a917c55f6ae733685de4c32ece678af16188f7f1" Sep 16 00:52:18.744965 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:52:18.744945 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e04ab29d012017eb5abe11e0a917c55f6ae733685de4c32ece678af16188f7f1\": container with ID starting with e04ab29d012017eb5abe11e0a917c55f6ae733685de4c32ece678af16188f7f1 not found: ID does not exist" containerID="e04ab29d012017eb5abe11e0a917c55f6ae733685de4c32ece678af16188f7f1" Sep 16 00:52:18.745019 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.744976 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04ab29d012017eb5abe11e0a917c55f6ae733685de4c32ece678af16188f7f1"} err="failed to get container status \"e04ab29d012017eb5abe11e0a917c55f6ae733685de4c32ece678af16188f7f1\": rpc error: code = NotFound desc = could not find container \"e04ab29d012017eb5abe11e0a917c55f6ae733685de4c32ece678af16188f7f1\": container with ID starting with e04ab29d012017eb5abe11e0a917c55f6ae733685de4c32ece678af16188f7f1 not found: ID does not exist" Sep 16 00:52:18.745019 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.745004 2567 scope.go:117] "RemoveContainer" containerID="3d9b25fdc72504276b6be424048bfdcff4481312e10e94a20a730af9ce5a039d" Sep 16 00:52:18.754736 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.754710 2567 scope.go:117] "RemoveContainer" containerID="3d9b25fdc72504276b6be424048bfdcff4481312e10e94a20a730af9ce5a039d" Sep 16 00:52:18.755172 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:52:18.755133 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d9b25fdc72504276b6be424048bfdcff4481312e10e94a20a730af9ce5a039d\": container with ID starting with 3d9b25fdc72504276b6be424048bfdcff4481312e10e94a20a730af9ce5a039d not found: ID does not exist" containerID="3d9b25fdc72504276b6be424048bfdcff4481312e10e94a20a730af9ce5a039d" Sep 16 00:52:18.755259 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.755192 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9b25fdc72504276b6be424048bfdcff4481312e10e94a20a730af9ce5a039d"} err="failed to get container status \"3d9b25fdc72504276b6be424048bfdcff4481312e10e94a20a730af9ce5a039d\": rpc error: code = NotFound desc = could not find container \"3d9b25fdc72504276b6be424048bfdcff4481312e10e94a20a730af9ce5a039d\": container with ID starting with 3d9b25fdc72504276b6be424048bfdcff4481312e10e94a20a730af9ce5a039d not found: ID does not exist" Sep 16 00:52:18.757341 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.757296 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh"] Sep 16 00:52:18.760531 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.760504 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-629e17ec-6597f464db-9s4zh"] Sep 16 00:52:18.776648 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.776610 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z"] Sep 16 00:52:18.779006 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.778983 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-629e17ec-5ffd4f4fb7-26f5z"] Sep 16 00:52:18.878265 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:18.878148 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:19.175540 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.175431 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5"] Sep 16 00:52:19.176072 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.176052 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6" containerName="revoke-keys" Sep 16 00:52:19.176161 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.176101 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6" containerName="revoke-keys" Sep 16 00:52:19.176161 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.176119 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ef9fff3-75e1-4e17-a195-19c513aee1cc" containerName="payload-pre-processing" Sep 16 00:52:19.176161 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.176126 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ef9fff3-75e1-4e17-a195-19c513aee1cc" containerName="payload-pre-processing" Sep 16 00:52:19.176161 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.176133 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d073679f-261b-42a6-9584-42927af437c5" containerName="payload-processing" Sep 16 00:52:19.176161 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.176158 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d073679f-261b-42a6-9584-42927af437c5" containerName="payload-processing" Sep 16 00:52:19.176418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.176368 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ef9fff3-75e1-4e17-a195-19c513aee1cc" containerName="payload-pre-processing" Sep 16 00:52:19.176418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.176389 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef5ea07-2db4-45b0-b4f8-a9d4abfc9cb6" containerName="revoke-keys" Sep 16 00:52:19.176418 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.176403 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d073679f-261b-42a6-9584-42927af437c5" containerName="payload-processing" Sep 16 00:52:19.181327 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.181298 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.183936 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.183913 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-69886ef2-gw-openshift-default-dockercfg-nw8jk\"" Sep 16 00:52:19.190488 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.190460 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5"] Sep 16 00:52:19.211633 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.211590 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-istio-data\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.211847 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.211821 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433b584b-025e-4354-8a54-db446ed01049-istio-token\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.211917 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.211883 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-credential-socket\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.212006 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.211943 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-workload-certs\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.212006 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.211989 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-istio-envoy\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.212125 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.212017 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433b584b-025e-4354-8a54-db446ed01049-istio-podinfo\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.212125 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.212103 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-workload-socket\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.212203 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.212143 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsc2v\" (UniqueName: \"kubernetes.io/projected/433b584b-025e-4354-8a54-db446ed01049-kube-api-access-zsc2v\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.212203 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.212167 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433b584b-025e-4354-8a54-db446ed01049-istiod-ca-cert\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.222945 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.222914 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:19.312964 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.312926 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433b584b-025e-4354-8a54-db446ed01049-istiod-ca-cert\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.313176 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.313086 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-istio-data\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.313176 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.313116 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433b584b-025e-4354-8a54-db446ed01049-istio-token\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.313176 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.313159 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-credential-socket\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.313361 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.313214 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-workload-certs\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.313361 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.313250 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-istio-envoy\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.313361 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.313279 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433b584b-025e-4354-8a54-db446ed01049-istio-podinfo\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.313361 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.313337 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-workload-socket\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.313575 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.313380 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsc2v\" (UniqueName: \"kubernetes.io/projected/433b584b-025e-4354-8a54-db446ed01049-kube-api-access-zsc2v\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.313636 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.313590 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-istio-data\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.313814 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.313766 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-workload-certs\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.313969 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.313888 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-credential-socket\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.314065 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.314021 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-workload-socket\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.314124 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.314069 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433b584b-025e-4354-8a54-db446ed01049-istiod-ca-cert\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.316240 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.316199 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-istio-envoy\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.316425 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.316406 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433b584b-025e-4354-8a54-db446ed01049-istio-podinfo\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.327912 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.327865 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433b584b-025e-4354-8a54-db446ed01049-istio-token\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.327912 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.327910 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsc2v\" (UniqueName: \"kubernetes.io/projected/433b584b-025e-4354-8a54-db446ed01049-kube-api-access-zsc2v\") pod \"e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.468372 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.468272 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:19.493939 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.493904 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:19.676866 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.676830 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:19.783188 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.783150 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5"] Sep 16 00:52:19.784640 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:52:19.784609 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433b584b_025e_4354_8a54_db446ed01049.slice/crio-c33d606389491077d5b98a392c217f0c5ca83135728f5d5fe72262c095f15432 WatchSource:0}: Error finding container c33d606389491077d5b98a392c217f0c5ca83135728f5d5fe72262c095f15432: Status 404 returned error can't find the container with id c33d606389491077d5b98a392c217f0c5ca83135728f5d5fe72262c095f15432 Sep 16 00:52:19.787504 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.787469 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:52:19.787634 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.787550 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:52:19.787634 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:19.787580 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:52:20.119129 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:20.119091 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ef9fff3-75e1-4e17-a195-19c513aee1cc" path="/var/lib/kubelet/pods/7ef9fff3-75e1-4e17-a195-19c513aee1cc/volumes" Sep 16 00:52:20.119516 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:20.119501 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d073679f-261b-42a6-9584-42927af437c5" path="/var/lib/kubelet/pods/d073679f-261b-42a6-9584-42927af437c5/volumes" Sep 16 00:52:20.540052 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:20.539943 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:20.757069 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:20.756984 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" event={"ID":"433b584b-025e-4354-8a54-db446ed01049","Type":"ContainerStarted","Data":"46f30ddaa1f70a414bc8772634ff2bf5028ada7ce1c5560660f1c5238df52834"} Sep 16 00:52:20.757287 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:20.757084 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" event={"ID":"433b584b-025e-4354-8a54-db446ed01049","Type":"ContainerStarted","Data":"c33d606389491077d5b98a392c217f0c5ca83135728f5d5fe72262c095f15432"} Sep 16 00:52:20.785028 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:20.784948 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" podStartSLOduration=1.7849226969999998 podStartE2EDuration="1.784922697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:52:20.777401638 +0000 UTC m=+1091.139625191" watchObservedRunningTime="2026-09-16 00:52:20.784922697 +0000 UTC m=+1091.147146226" Sep 16 00:52:21.494400 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:21.494356 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:21.499816 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:21.499787 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:21.622451 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:21.622419 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:21.724981 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:21.724945 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:21.763151 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:21.763052 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:21.764164 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:21.764145 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:22.033441 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:22.033350 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:22.256549 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:22.256515 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:22.355890 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:22.355852 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:22.360581 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:22.360546 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22"] Sep 16 00:52:22.365403 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:22.365375 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22" Sep 16 00:52:22.367975 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:22.367950 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-2fa86d\"" Sep 16 00:52:22.368161 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:22.368016 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-2fa86d-dockercfg-kd4lp\"" Sep 16 00:52:22.374142 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:22.373620 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22"] Sep 16 00:52:22.448198 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:22.448160 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xntmx\" (UniqueName: \"kubernetes.io/projected/15d05ebf-2444-4d61-9e69-94afc252b036-kube-api-access-xntmx\") pod \"payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22\" (UID: \"15d05ebf-2444-4d61-9e69-94afc252b036\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22" Sep 16 00:52:22.448393 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:22.448242 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d05ebf-2444-4d61-9e69-94afc252b036-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22\" (UID: \"15d05ebf-2444-4d61-9e69-94afc252b036\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22" Sep 16 00:52:22.549453 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:22.549409 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xntmx\" (UniqueName: \"kubernetes.io/projected/15d05ebf-2444-4d61-9e69-94afc252b036-kube-api-access-xntmx\") pod \"payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22\" (UID: \"15d05ebf-2444-4d61-9e69-94afc252b036\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22" Sep 16 00:52:22.549675 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:22.549473 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d05ebf-2444-4d61-9e69-94afc252b036-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22\" (UID: \"15d05ebf-2444-4d61-9e69-94afc252b036\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22" Sep 16 00:52:22.550320 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:22.550299 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d05ebf-2444-4d61-9e69-94afc252b036-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22\" (UID: \"15d05ebf-2444-4d61-9e69-94afc252b036\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22" Sep 16 00:52:22.557750 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:22.557716 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xntmx\" (UniqueName: \"kubernetes.io/projected/15d05ebf-2444-4d61-9e69-94afc252b036-kube-api-access-xntmx\") pod \"payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22\" (UID: \"15d05ebf-2444-4d61-9e69-94afc252b036\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22" Sep 16 00:52:22.678914 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:22.678813 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22" Sep 16 00:52:22.847028 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:22.846991 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:22.961089 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:22.961057 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22"] Sep 16 00:52:22.963336 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:52:22.963298 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d05ebf_2444_4d61_9e69_94afc252b036.slice/crio-2eb5752505504e86e1593a67af2ff56e47b425ae4e28d91c00f2b803069df81d WatchSource:0}: Error finding container 2eb5752505504e86e1593a67af2ff56e47b425ae4e28d91c00f2b803069df81d: Status 404 returned error can't find the container with id 2eb5752505504e86e1593a67af2ff56e47b425ae4e28d91c00f2b803069df81d Sep 16 00:52:23.718989 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:23.718955 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:23.777965 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:23.777930 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22" event={"ID":"15d05ebf-2444-4d61-9e69-94afc252b036","Type":"ContainerStarted","Data":"194f1b24443d78ee54ea3aa60782fb101f454e6a98b4f37ed498783d935af804"} Sep 16 00:52:23.778181 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:23.777988 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22" event={"ID":"15d05ebf-2444-4d61-9e69-94afc252b036","Type":"ContainerStarted","Data":"2eb5752505504e86e1593a67af2ff56e47b425ae4e28d91c00f2b803069df81d"} Sep 16 00:52:23.778181 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:23.778010 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22" Sep 16 00:52:23.798671 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:23.798612 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22" podStartSLOduration=1.798583953 podStartE2EDuration="1.798583953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:52:23.795983213 +0000 UTC m=+1094.158206769" watchObservedRunningTime="2026-09-16 00:52:23.798583953 +0000 UTC m=+1094.160807488" Sep 16 00:52:24.599162 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:24.599131 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:24.698074 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:24.698010 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:25.032970 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:25.032882 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:25.121897 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:25.121866 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:25.174009 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:25.173969 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5"] Sep 16 00:52:25.177781 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:25.177758 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5" Sep 16 00:52:25.180536 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:25.180507 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-69886ef2\"" Sep 16 00:52:25.180684 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:25.180575 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-69886ef2-dockercfg-tlktp\"" Sep 16 00:52:25.189444 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:25.188333 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5"] Sep 16 00:52:25.220399 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:25.220364 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:25.277570 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:25.277523 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzcdg\" (UniqueName: \"kubernetes.io/projected/661896ad-bbe7-4063-91fb-2fb83a946d74-kube-api-access-xzcdg\") pod \"payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5\" (UID: \"661896ad-bbe7-4063-91fb-2fb83a946d74\") " pod="openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5" Sep 16 00:52:25.277761 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:25.277628 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/661896ad-bbe7-4063-91fb-2fb83a946d74-plugins-config-volume\") pod \"payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5\" (UID: \"661896ad-bbe7-4063-91fb-2fb83a946d74\") " pod="openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5" Sep 16 00:52:25.378716 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:25.378662 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/661896ad-bbe7-4063-91fb-2fb83a946d74-plugins-config-volume\") pod \"payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5\" (UID: \"661896ad-bbe7-4063-91fb-2fb83a946d74\") " pod="openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5" Sep 16 00:52:25.378901 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:25.378786 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzcdg\" (UniqueName: \"kubernetes.io/projected/661896ad-bbe7-4063-91fb-2fb83a946d74-kube-api-access-xzcdg\") pod \"payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5\" (UID: \"661896ad-bbe7-4063-91fb-2fb83a946d74\") " pod="openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5" Sep 16 00:52:25.379477 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:25.379457 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/661896ad-bbe7-4063-91fb-2fb83a946d74-plugins-config-volume\") pod \"payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5\" (UID: \"661896ad-bbe7-4063-91fb-2fb83a946d74\") " pod="openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5" Sep 16 00:52:25.387058 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:25.387012 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzcdg\" (UniqueName: \"kubernetes.io/projected/661896ad-bbe7-4063-91fb-2fb83a946d74-kube-api-access-xzcdg\") pod \"payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5\" (UID: \"661896ad-bbe7-4063-91fb-2fb83a946d74\") " pod="openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5" Sep 16 00:52:25.499773 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:25.499730 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5" Sep 16 00:52:25.549538 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:25.549503 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:25.778437 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:25.778407 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5"] Sep 16 00:52:25.780747 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:52:25.780712 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661896ad_bbe7_4063_91fb_2fb83a946d74.slice/crio-5513ed8142c1dc46b69feb9041e7bc701270659beb530d2575558175b2c04411 WatchSource:0}: Error finding container 5513ed8142c1dc46b69feb9041e7bc701270659beb530d2575558175b2c04411: Status 404 returned error can't find the container with id 5513ed8142c1dc46b69feb9041e7bc701270659beb530d2575558175b2c04411 Sep 16 00:52:25.791900 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:25.791870 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5" event={"ID":"661896ad-bbe7-4063-91fb-2fb83a946d74","Type":"ContainerStarted","Data":"5513ed8142c1dc46b69feb9041e7bc701270659beb530d2575558175b2c04411"} Sep 16 00:52:26.522547 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:26.522503 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:26.801943 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:26.801900 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5" event={"ID":"661896ad-bbe7-4063-91fb-2fb83a946d74","Type":"ContainerStarted","Data":"f86b104df34140b18ee52f83bd1ab44e4e7e109433bb3e8123d5399f5d8e3e5d"} Sep 16 00:52:26.802163 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:26.802030 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5" Sep 16 00:52:26.820894 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:26.820835 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5" podStartSLOduration=1.820817277 podStartE2EDuration="1.820817277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:52:26.817592881 +0000 UTC m=+1097.179816497" watchObservedRunningTime="2026-09-16 00:52:26.820817277 +0000 UTC m=+1097.183040809" Sep 16 00:52:27.151214 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:27.151122 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:27.783360 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:27.783325 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:33.916585 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:33.916546 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:34.786839 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:34.786803 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22" Sep 16 00:52:37.810438 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:37.810397 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5" Sep 16 00:52:42.915536 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:42.915480 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:44.333738 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:44.333695 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:45.088288 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:45.088241 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:45.719471 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:45.719435 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:46.246765 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:46.245777 2567 patch_prober.go:28] interesting pod/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:38498->10.134.0.53:15021: read: connection reset by peer" start-of-body= Sep 16 00:52:46.246765 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:46.245996 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" podUID="0f89837a-e1cf-4d80-a25c-145602b37ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:38498->10.134.0.53:15021: read: connection reset by peer" Sep 16 00:52:46.443477 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:46.443375 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:46.887557 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:46.887530 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:46.979258 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:46.972986 2567 generic.go:356] "Generic (PLEG): container finished" podID="0f89837a-e1cf-4d80-a25c-145602b37ca7" containerID="18e6ebca4deef2ef900e844b5909b463b99da48dcd0879cf816874aedd447665" exitCode=0 Sep 16 00:52:46.979258 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:46.973164 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" event={"ID":"0f89837a-e1cf-4d80-a25c-145602b37ca7","Type":"ContainerDied","Data":"18e6ebca4deef2ef900e844b5909b463b99da48dcd0879cf816874aedd447665"} Sep 16 00:52:46.979258 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:46.973752 2567 scope.go:117] "RemoveContainer" containerID="18e6ebca4deef2ef900e844b5909b463b99da48dcd0879cf816874aedd447665" Sep 16 00:52:46.979258 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:46.975855 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:52:46.979258 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:46.975951 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:52:46.979258 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:46.975997 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:52:46.985985 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:46.985947 2567 generic.go:356] "Generic (PLEG): container finished" podID="433b584b-025e-4354-8a54-db446ed01049" containerID="46f30ddaa1f70a414bc8772634ff2bf5028ada7ce1c5560660f1c5238df52834" exitCode=0 Sep 16 00:52:46.986165 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:46.986022 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" event={"ID":"433b584b-025e-4354-8a54-db446ed01049","Type":"ContainerDied","Data":"46f30ddaa1f70a414bc8772634ff2bf5028ada7ce1c5560660f1c5238df52834"} Sep 16 00:52:46.986557 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:46.986540 2567 scope.go:117] "RemoveContainer" containerID="46f30ddaa1f70a414bc8772634ff2bf5028ada7ce1c5560660f1c5238df52834" Sep 16 00:52:46.990001 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:46.989957 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:52:46.990144 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:46.990086 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:52:46.990204 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:46.990180 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:52:47.133127 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:47.133012 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:47.924314 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:47.924283 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:48.008948 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:48.008905 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" event={"ID":"0f89837a-e1cf-4d80-a25c-145602b37ca7","Type":"ContainerStarted","Data":"7d9e537a3759b4a43855cecfdd4d64d594cb54e233b7d2e318fcaa1338553637"} Sep 16 00:52:48.011936 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:48.011892 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" event={"ID":"433b584b-025e-4354-8a54-db446ed01049","Type":"ContainerStarted","Data":"b06b07e4ddcfd64aa018a99c2a8a6f82fc59857bf725962256178df79803f606"} Sep 16 00:52:48.120110 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:48.120073 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:48.120381 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:48.120358 2567 patch_prober.go:28] interesting pod/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 16 00:52:48.120431 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:48.120409 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" podUID="0f89837a-e1cf-4d80-a25c-145602b37ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 16 00:52:48.494948 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:48.494846 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:48.496676 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:48.496392 2567 patch_prober.go:28] interesting pod/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 16 00:52:48.496676 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:48.496443 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" podUID="433b584b-025e-4354-8a54-db446ed01049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 16 00:52:48.698122 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:48.698082 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:49.120835 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:49.120802 2567 patch_prober.go:28] interesting pod/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 16 00:52:49.121324 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:49.120856 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" podUID="0f89837a-e1cf-4d80-a25c-145602b37ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 16 00:52:49.494891 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:49.494773 2567 patch_prober.go:28] interesting pod/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 16 00:52:49.494891 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:49.494874 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" podUID="433b584b-025e-4354-8a54-db446ed01049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 16 00:52:49.524412 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:49.524366 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:50.120338 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:50.120295 2567 patch_prober.go:28] interesting pod/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 16 00:52:50.120546 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:50.120361 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" podUID="0f89837a-e1cf-4d80-a25c-145602b37ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 16 00:52:50.333297 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:50.333265 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:50.494736 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:50.494617 2567 patch_prober.go:28] interesting pod/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 16 00:52:50.494736 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:50.494697 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" podUID="433b584b-025e-4354-8a54-db446ed01049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 16 00:52:50.965784 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:50.965738 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:51.120758 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:51.120717 2567 patch_prober.go:28] interesting pod/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 16 00:52:51.120947 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:51.120775 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" podUID="0f89837a-e1cf-4d80-a25c-145602b37ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 16 00:52:51.494994 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:51.494950 2567 patch_prober.go:28] interesting pod/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 16 00:52:51.495440 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:51.495012 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" podUID="433b584b-025e-4354-8a54-db446ed01049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 16 00:52:51.763690 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:51.763577 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:52.047137 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:52.046558 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:52:52.120863 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:52.120820 2567 patch_prober.go:28] interesting pod/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 16 00:52:52.121096 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:52.120892 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" podUID="0f89837a-e1cf-4d80-a25c-145602b37ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 16 00:52:52.494526 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:52.494484 2567 patch_prober.go:28] interesting pod/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 16 00:52:52.494952 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:52.494913 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" podUID="433b584b-025e-4354-8a54-db446ed01049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 16 00:52:52.767798 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:52.767686 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:53.120751 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:53.120713 2567 patch_prober.go:28] interesting pod/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 16 00:52:53.121067 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:53.121013 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" podUID="0f89837a-e1cf-4d80-a25c-145602b37ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 16 00:52:53.427508 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:53.427408 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:53.495085 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:53.495028 2567 patch_prober.go:28] interesting pod/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 16 00:52:53.495464 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:53.495429 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" podUID="433b584b-025e-4354-8a54-db446ed01049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 16 00:52:54.124587 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:54.124556 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:54.125096 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:54.124838 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:54.125581 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:54.125561 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 00:52:54.322324 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:54.322291 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:54.499618 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:54.499520 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:54.500790 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:54.500764 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:52:55.238674 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:55.238640 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:55.983205 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:55.983165 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:52:56.962005 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:52:56.961974 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:02.021940 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:02.021900 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:02.738673 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:02.738622 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:03.392514 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:03.392477 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:03.409998 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:03.409950 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5"] Sep 16 00:53:03.410262 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:03.410231 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5" podUID="661896ad-bbe7-4063-91fb-2fb83a946d74" containerName="payload-processing" containerID="cri-o://f86b104df34140b18ee52f83bd1ab44e4e7e109433bb3e8123d5399f5d8e3e5d" gracePeriod=30 Sep 16 00:53:03.492868 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:03.492829 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:03.591526 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:03.591400 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:03.649746 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:03.649665 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:03.680340 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:03.680306 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5" Sep 16 00:53:03.729443 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:03.729410 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5"] Sep 16 00:53:03.729709 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:03.729671 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" podUID="433b584b-025e-4354-8a54-db446ed01049" containerName="istio-proxy" containerID="cri-o://b06b07e4ddcfd64aa018a99c2a8a6f82fc59857bf725962256178df79803f606" gracePeriod=30 Sep 16 00:53:03.736654 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:03.736620 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/661896ad-bbe7-4063-91fb-2fb83a946d74-plugins-config-volume\") pod \"661896ad-bbe7-4063-91fb-2fb83a946d74\" (UID: \"661896ad-bbe7-4063-91fb-2fb83a946d74\") " Sep 16 00:53:03.736861 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:03.736816 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzcdg\" (UniqueName: \"kubernetes.io/projected/661896ad-bbe7-4063-91fb-2fb83a946d74-kube-api-access-xzcdg\") pod \"661896ad-bbe7-4063-91fb-2fb83a946d74\" (UID: \"661896ad-bbe7-4063-91fb-2fb83a946d74\") " Sep 16 00:53:03.737109 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:03.737063 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661896ad-bbe7-4063-91fb-2fb83a946d74-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "661896ad-bbe7-4063-91fb-2fb83a946d74" (UID: "661896ad-bbe7-4063-91fb-2fb83a946d74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:53:03.737267 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:03.737222 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/661896ad-bbe7-4063-91fb-2fb83a946d74-plugins-config-volume\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:03.740050 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:03.739990 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661896ad-bbe7-4063-91fb-2fb83a946d74-kube-api-access-xzcdg" (OuterVolumeSpecName: "kube-api-access-xzcdg") pod "661896ad-bbe7-4063-91fb-2fb83a946d74" (UID: "661896ad-bbe7-4063-91fb-2fb83a946d74"). InnerVolumeSpecName "kube-api-access-xzcdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:53:03.747709 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:03.747676 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:03.838143 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:03.838100 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzcdg\" (UniqueName: \"kubernetes.io/projected/661896ad-bbe7-4063-91fb-2fb83a946d74-kube-api-access-xzcdg\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:03.996455 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:03.996360 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:04.120065 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:04.119995 2567 generic.go:356] "Generic (PLEG): container finished" podID="661896ad-bbe7-4063-91fb-2fb83a946d74" containerID="f86b104df34140b18ee52f83bd1ab44e4e7e109433bb3e8123d5399f5d8e3e5d" exitCode=0 Sep 16 00:53:04.120248 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:04.120082 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5" Sep 16 00:53:04.120248 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:04.120093 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5" event={"ID":"661896ad-bbe7-4063-91fb-2fb83a946d74","Type":"ContainerDied","Data":"f86b104df34140b18ee52f83bd1ab44e4e7e109433bb3e8123d5399f5d8e3e5d"} Sep 16 00:53:04.120248 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:04.120159 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5" event={"ID":"661896ad-bbe7-4063-91fb-2fb83a946d74","Type":"ContainerDied","Data":"5513ed8142c1dc46b69feb9041e7bc701270659beb530d2575558175b2c04411"} Sep 16 00:53:04.120248 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:04.120184 2567 scope.go:117] "RemoveContainer" containerID="f86b104df34140b18ee52f83bd1ab44e4e7e109433bb3e8123d5399f5d8e3e5d" Sep 16 00:53:04.131909 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:04.131883 2567 scope.go:117] "RemoveContainer" containerID="f86b104df34140b18ee52f83bd1ab44e4e7e109433bb3e8123d5399f5d8e3e5d" Sep 16 00:53:04.132367 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:53:04.132343 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f86b104df34140b18ee52f83bd1ab44e4e7e109433bb3e8123d5399f5d8e3e5d\": container with ID starting with f86b104df34140b18ee52f83bd1ab44e4e7e109433bb3e8123d5399f5d8e3e5d not found: ID does not exist" containerID="f86b104df34140b18ee52f83bd1ab44e4e7e109433bb3e8123d5399f5d8e3e5d" Sep 16 00:53:04.132456 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:04.132379 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f86b104df34140b18ee52f83bd1ab44e4e7e109433bb3e8123d5399f5d8e3e5d"} err="failed to get container status \"f86b104df34140b18ee52f83bd1ab44e4e7e109433bb3e8123d5399f5d8e3e5d\": rpc error: code = NotFound desc = could not find container \"f86b104df34140b18ee52f83bd1ab44e4e7e109433bb3e8123d5399f5d8e3e5d\": container with ID starting with f86b104df34140b18ee52f83bd1ab44e4e7e109433bb3e8123d5399f5d8e3e5d not found: ID does not exist" Sep 16 00:53:04.147508 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:04.147442 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5"] Sep 16 00:53:04.150358 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:04.150331 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-69886ef2-58b45b4d86-cmhs5"] Sep 16 00:53:04.200236 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:04.200203 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:04.535539 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:04.535508 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:04.843601 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:04.843571 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:05.052436 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:05.052406 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:05.720741 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:05.720703 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:06.119800 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:06.119761 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="661896ad-bbe7-4063-91fb-2fb83a946d74" path="/var/lib/kubelet/pods/661896ad-bbe7-4063-91fb-2fb83a946d74/volumes" Sep 16 00:53:06.507470 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:06.507376 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:07.021875 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:07.021830 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:07.123266 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:07.123228 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:07.641606 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:07.641565 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:08.420647 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:08.420604 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:09.092780 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.092753 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:53:09.155868 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.155776 2567 generic.go:356] "Generic (PLEG): container finished" podID="433b584b-025e-4354-8a54-db446ed01049" containerID="b06b07e4ddcfd64aa018a99c2a8a6f82fc59857bf725962256178df79803f606" exitCode=0 Sep 16 00:53:09.155868 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.155825 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" Sep 16 00:53:09.156116 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.155865 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" event={"ID":"433b584b-025e-4354-8a54-db446ed01049","Type":"ContainerDied","Data":"b06b07e4ddcfd64aa018a99c2a8a6f82fc59857bf725962256178df79803f606"} Sep 16 00:53:09.156116 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.155924 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5" event={"ID":"433b584b-025e-4354-8a54-db446ed01049","Type":"ContainerDied","Data":"c33d606389491077d5b98a392c217f0c5ca83135728f5d5fe72262c095f15432"} Sep 16 00:53:09.156116 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.155946 2567 scope.go:117] "RemoveContainer" containerID="b06b07e4ddcfd64aa018a99c2a8a6f82fc59857bf725962256178df79803f606" Sep 16 00:53:09.166211 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.166184 2567 scope.go:117] "RemoveContainer" containerID="46f30ddaa1f70a414bc8772634ff2bf5028ada7ce1c5560660f1c5238df52834" Sep 16 00:53:09.177124 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.177096 2567 scope.go:117] "RemoveContainer" containerID="b06b07e4ddcfd64aa018a99c2a8a6f82fc59857bf725962256178df79803f606" Sep 16 00:53:09.177512 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:53:09.177491 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b06b07e4ddcfd64aa018a99c2a8a6f82fc59857bf725962256178df79803f606\": container with ID starting with b06b07e4ddcfd64aa018a99c2a8a6f82fc59857bf725962256178df79803f606 not found: ID does not exist" containerID="b06b07e4ddcfd64aa018a99c2a8a6f82fc59857bf725962256178df79803f606" Sep 16 00:53:09.177562 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.177526 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06b07e4ddcfd64aa018a99c2a8a6f82fc59857bf725962256178df79803f606"} err="failed to get container status \"b06b07e4ddcfd64aa018a99c2a8a6f82fc59857bf725962256178df79803f606\": rpc error: code = NotFound desc = could not find container \"b06b07e4ddcfd64aa018a99c2a8a6f82fc59857bf725962256178df79803f606\": container with ID starting with b06b07e4ddcfd64aa018a99c2a8a6f82fc59857bf725962256178df79803f606 not found: ID does not exist" Sep 16 00:53:09.177562 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.177548 2567 scope.go:117] "RemoveContainer" containerID="46f30ddaa1f70a414bc8772634ff2bf5028ada7ce1c5560660f1c5238df52834" Sep 16 00:53:09.177903 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:53:09.177884 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f30ddaa1f70a414bc8772634ff2bf5028ada7ce1c5560660f1c5238df52834\": container with ID starting with 46f30ddaa1f70a414bc8772634ff2bf5028ada7ce1c5560660f1c5238df52834 not found: ID does not exist" containerID="46f30ddaa1f70a414bc8772634ff2bf5028ada7ce1c5560660f1c5238df52834" Sep 16 00:53:09.177961 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.177910 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f30ddaa1f70a414bc8772634ff2bf5028ada7ce1c5560660f1c5238df52834"} err="failed to get container status \"46f30ddaa1f70a414bc8772634ff2bf5028ada7ce1c5560660f1c5238df52834\": rpc error: code = NotFound desc = could not find container \"46f30ddaa1f70a414bc8772634ff2bf5028ada7ce1c5560660f1c5238df52834\": container with ID starting with 46f30ddaa1f70a414bc8772634ff2bf5028ada7ce1c5560660f1c5238df52834 not found: ID does not exist" Sep 16 00:53:09.197343 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.197301 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-istio-data\") pod \"433b584b-025e-4354-8a54-db446ed01049\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " Sep 16 00:53:09.197343 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.197343 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-istio-envoy\") pod \"433b584b-025e-4354-8a54-db446ed01049\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " Sep 16 00:53:09.197614 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.197384 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433b584b-025e-4354-8a54-db446ed01049-istiod-ca-cert\") pod \"433b584b-025e-4354-8a54-db446ed01049\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " Sep 16 00:53:09.197614 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.197415 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433b584b-025e-4354-8a54-db446ed01049-istio-token\") pod \"433b584b-025e-4354-8a54-db446ed01049\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " Sep 16 00:53:09.197614 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.197460 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-workload-socket\") pod \"433b584b-025e-4354-8a54-db446ed01049\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " Sep 16 00:53:09.197614 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.197491 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-credential-socket\") pod \"433b584b-025e-4354-8a54-db446ed01049\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " Sep 16 00:53:09.197614 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.197538 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-workload-certs\") pod \"433b584b-025e-4354-8a54-db446ed01049\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " Sep 16 00:53:09.197614 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.197582 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsc2v\" (UniqueName: \"kubernetes.io/projected/433b584b-025e-4354-8a54-db446ed01049-kube-api-access-zsc2v\") pod \"433b584b-025e-4354-8a54-db446ed01049\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " Sep 16 00:53:09.197930 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.197667 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433b584b-025e-4354-8a54-db446ed01049-istio-podinfo\") pod \"433b584b-025e-4354-8a54-db446ed01049\" (UID: \"433b584b-025e-4354-8a54-db446ed01049\") " Sep 16 00:53:09.197930 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.197760 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "433b584b-025e-4354-8a54-db446ed01049" (UID: "433b584b-025e-4354-8a54-db446ed01049"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:53:09.197930 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.197857 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "433b584b-025e-4354-8a54-db446ed01049" (UID: "433b584b-025e-4354-8a54-db446ed01049"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:53:09.197930 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.197868 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433b584b-025e-4354-8a54-db446ed01049-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "433b584b-025e-4354-8a54-db446ed01049" (UID: "433b584b-025e-4354-8a54-db446ed01049"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:53:09.197930 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.197868 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "433b584b-025e-4354-8a54-db446ed01049" (UID: "433b584b-025e-4354-8a54-db446ed01049"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:53:09.197930 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.197650 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-istio-data" (OuterVolumeSpecName: "istio-data") pod "433b584b-025e-4354-8a54-db446ed01049" (UID: "433b584b-025e-4354-8a54-db446ed01049"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:53:09.198210 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.198090 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-istio-data\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:09.198210 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.198112 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433b584b-025e-4354-8a54-db446ed01049-istiod-ca-cert\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:09.198210 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.198126 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-workload-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:09.198210 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.198140 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-credential-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:09.198210 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.198154 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-workload-certs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:09.199991 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.199963 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433b584b-025e-4354-8a54-db446ed01049-istio-token" (OuterVolumeSpecName: "istio-token") pod "433b584b-025e-4354-8a54-db446ed01049" (UID: "433b584b-025e-4354-8a54-db446ed01049"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:53:09.200164 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.200010 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "433b584b-025e-4354-8a54-db446ed01049" (UID: "433b584b-025e-4354-8a54-db446ed01049"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:53:09.200164 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.200009 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433b584b-025e-4354-8a54-db446ed01049-kube-api-access-zsc2v" (OuterVolumeSpecName: "kube-api-access-zsc2v") pod "433b584b-025e-4354-8a54-db446ed01049" (UID: "433b584b-025e-4354-8a54-db446ed01049"). InnerVolumeSpecName "kube-api-access-zsc2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:53:09.200164 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.200122 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/433b584b-025e-4354-8a54-db446ed01049-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "433b584b-025e-4354-8a54-db446ed01049" (UID: "433b584b-025e-4354-8a54-db446ed01049"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 00:53:09.218524 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.218489 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:09.298834 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.298798 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433b584b-025e-4354-8a54-db446ed01049-istio-podinfo\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:09.298834 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.298830 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433b584b-025e-4354-8a54-db446ed01049-istio-envoy\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:09.298834 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.298840 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433b584b-025e-4354-8a54-db446ed01049-istio-token\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:09.299137 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.298849 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsc2v\" (UniqueName: \"kubernetes.io/projected/433b584b-025e-4354-8a54-db446ed01049-kube-api-access-zsc2v\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:09.481370 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.481334 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5"] Sep 16 00:53:09.483815 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.483789 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-69886ef2-gw-openshift-default-7dc88d95b8-vcxt5"] Sep 16 00:53:09.940762 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:09.940726 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:10.041224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.041190 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:10.125605 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.125566 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="433b584b-025e-4354-8a54-db446ed01049" path="/var/lib/kubelet/pods/433b584b-025e-4354-8a54-db446ed01049/volumes" Sep 16 00:53:10.316314 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.316277 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:10.417286 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.417254 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:10.516723 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.516691 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:10.529101 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.529065 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22"] Sep 16 00:53:10.529489 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.529475 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="661896ad-bbe7-4063-91fb-2fb83a946d74" containerName="payload-processing" Sep 16 00:53:10.529534 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.529490 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="661896ad-bbe7-4063-91fb-2fb83a946d74" containerName="payload-processing" Sep 16 00:53:10.529534 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.529500 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="433b584b-025e-4354-8a54-db446ed01049" containerName="istio-proxy" Sep 16 00:53:10.529534 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.529506 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="433b584b-025e-4354-8a54-db446ed01049" containerName="istio-proxy" Sep 16 00:53:10.529534 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.529511 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="433b584b-025e-4354-8a54-db446ed01049" containerName="istio-proxy" Sep 16 00:53:10.529534 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.529516 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="433b584b-025e-4354-8a54-db446ed01049" containerName="istio-proxy" Sep 16 00:53:10.529686 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.529568 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="661896ad-bbe7-4063-91fb-2fb83a946d74" containerName="payload-processing" Sep 16 00:53:10.529686 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.529575 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="433b584b-025e-4354-8a54-db446ed01049" containerName="istio-proxy" Sep 16 00:53:10.529686 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.529584 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="433b584b-025e-4354-8a54-db446ed01049" containerName="istio-proxy" Sep 16 00:53:10.532997 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.532962 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22" Sep 16 00:53:10.535768 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.535740 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9f35fa48-dockercfg-fl6n9\"" Sep 16 00:53:10.535914 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.535786 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9f35fa48\"" Sep 16 00:53:10.543217 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.543189 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22"] Sep 16 00:53:10.613840 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.613728 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c56d99-0337-4bb5-8011-83d3b612a6d7-plugins-config-volume\") pod \"payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22\" (UID: \"20c56d99-0337-4bb5-8011-83d3b612a6d7\") " pod="openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22" Sep 16 00:53:10.613840 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.613815 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dplmk\" (UniqueName: \"kubernetes.io/projected/20c56d99-0337-4bb5-8011-83d3b612a6d7-kube-api-access-dplmk\") pod \"payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22\" (UID: \"20c56d99-0337-4bb5-8011-83d3b612a6d7\") " pod="openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22" Sep 16 00:53:10.714824 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.714769 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c56d99-0337-4bb5-8011-83d3b612a6d7-plugins-config-volume\") pod \"payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22\" (UID: \"20c56d99-0337-4bb5-8011-83d3b612a6d7\") " pod="openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22" Sep 16 00:53:10.714824 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.714838 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dplmk\" (UniqueName: \"kubernetes.io/projected/20c56d99-0337-4bb5-8011-83d3b612a6d7-kube-api-access-dplmk\") pod \"payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22\" (UID: \"20c56d99-0337-4bb5-8011-83d3b612a6d7\") " pod="openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22" Sep 16 00:53:10.715555 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.715534 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c56d99-0337-4bb5-8011-83d3b612a6d7-plugins-config-volume\") pod \"payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22\" (UID: \"20c56d99-0337-4bb5-8011-83d3b612a6d7\") " pod="openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22" Sep 16 00:53:10.723581 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.723546 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dplmk\" (UniqueName: \"kubernetes.io/projected/20c56d99-0337-4bb5-8011-83d3b612a6d7-kube-api-access-dplmk\") pod \"payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22\" (UID: \"20c56d99-0337-4bb5-8011-83d3b612a6d7\") " pod="openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22" Sep 16 00:53:10.844823 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:10.844779 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22" Sep 16 00:53:11.121686 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:11.121652 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22"] Sep 16 00:53:11.123527 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:53:11.123496 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c56d99_0337_4bb5_8011_83d3b612a6d7.slice/crio-e7a0d3da7c58cfb746b67201543f662107b9e96a1bf14416c0c56665beb07d6a WatchSource:0}: Error finding container e7a0d3da7c58cfb746b67201543f662107b9e96a1bf14416c0c56665beb07d6a: Status 404 returned error can't find the container with id e7a0d3da7c58cfb746b67201543f662107b9e96a1bf14416c0c56665beb07d6a Sep 16 00:53:11.175284 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:11.175252 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22" event={"ID":"20c56d99-0337-4bb5-8011-83d3b612a6d7","Type":"ContainerStarted","Data":"e7a0d3da7c58cfb746b67201543f662107b9e96a1bf14416c0c56665beb07d6a"} Sep 16 00:53:11.876168 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:11.876123 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:11.991747 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:11.991712 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f7c3b-c5wcs"] Sep 16 00:53:11.994951 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:11.994928 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f7c3b-c5wcs" Sep 16 00:53:11.998622 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:11.998590 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-npltg\"" Sep 16 00:53:11.998622 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:11.998608 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 00:53:12.001282 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:12.001247 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f7c3b-c5wcs"] Sep 16 00:53:12.129526 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:12.129429 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbe1377-d9f4-4038-98ba-70a1b5c6da2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0f7c3b-c5wcs\" (UID: \"dcbe1377-d9f4-4038-98ba-70a1b5c6da2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f7c3b-c5wcs" Sep 16 00:53:12.184252 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:12.184211 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22" event={"ID":"20c56d99-0337-4bb5-8011-83d3b612a6d7","Type":"ContainerStarted","Data":"fd99e2b5cb447bad71991a2804487fb4a503e8dc46c84612432afd14ddaf9e25"} Sep 16 00:53:12.184457 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:12.184296 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22" Sep 16 00:53:12.205267 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:12.205201 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22" podStartSLOduration=2.205179786 podStartE2EDuration="2.205179786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:53:12.202250078 +0000 UTC m=+1142.564473633" watchObservedRunningTime="2026-09-16 00:53:12.205179786 +0000 UTC m=+1142.567403318" Sep 16 00:53:12.230741 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:12.230698 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbe1377-d9f4-4038-98ba-70a1b5c6da2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0f7c3b-c5wcs\" (UID: \"dcbe1377-d9f4-4038-98ba-70a1b5c6da2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f7c3b-c5wcs" Sep 16 00:53:12.231445 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:12.231425 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbe1377-d9f4-4038-98ba-70a1b5c6da2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0f7c3b-c5wcs\" (UID: \"dcbe1377-d9f4-4038-98ba-70a1b5c6da2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f7c3b-c5wcs" Sep 16 00:53:12.310222 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:12.310186 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f7c3b-c5wcs" Sep 16 00:53:12.334690 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:12.334638 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:12.612931 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:12.612901 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f7c3b-c5wcs"] Sep 16 00:53:12.614955 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:53:12.614924 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbe1377_d9f4_4038_98ba_70a1b5c6da2c.slice/crio-5c3d9d237faca824f2d18fc233b01ec9b813d978b55fd090f72533ebba3c8c06 WatchSource:0}: Error finding container 5c3d9d237faca824f2d18fc233b01ec9b813d978b55fd090f72533ebba3c8c06: Status 404 returned error can't find the container with id 5c3d9d237faca824f2d18fc233b01ec9b813d978b55fd090f72533ebba3c8c06 Sep 16 00:53:13.193648 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:13.193593 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f7c3b-c5wcs" event={"ID":"dcbe1377-d9f4-4038-98ba-70a1b5c6da2c","Type":"ContainerStarted","Data":"911d7ef5534e5a0b3c9fe01dafcc403541c8d20a5de9ab306cbeff63541573a3"} Sep 16 00:53:13.194226 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:13.193671 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f7c3b-c5wcs" event={"ID":"dcbe1377-d9f4-4038-98ba-70a1b5c6da2c","Type":"ContainerStarted","Data":"5c3d9d237faca824f2d18fc233b01ec9b813d978b55fd090f72533ebba3c8c06"} Sep 16 00:53:13.210293 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:13.210183 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f7c3b-c5wcs" podStartSLOduration=2.210164356 podStartE2EDuration="2.210164356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:53:13.207215904 +0000 UTC m=+1143.569439457" watchObservedRunningTime="2026-09-16 00:53:13.210164356 +0000 UTC m=+1143.572387940" Sep 16 00:53:13.226029 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:13.225991 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:13.893824 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:13.893785 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:14.202016 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:14.201918 2567 generic.go:356] "Generic (PLEG): container finished" podID="dcbe1377-d9f4-4038-98ba-70a1b5c6da2c" containerID="911d7ef5534e5a0b3c9fe01dafcc403541c8d20a5de9ab306cbeff63541573a3" exitCode=0 Sep 16 00:53:14.202016 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:14.201958 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f7c3b-c5wcs" event={"ID":"dcbe1377-d9f4-4038-98ba-70a1b5c6da2c","Type":"ContainerDied","Data":"911d7ef5534e5a0b3c9fe01dafcc403541c8d20a5de9ab306cbeff63541573a3"} Sep 16 00:53:14.628051 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:14.628017 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:15.271629 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:15.271587 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:15.360950 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:15.360922 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f7c3b-c5wcs" Sep 16 00:53:15.466360 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:15.466257 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbe1377-d9f4-4038-98ba-70a1b5c6da2c-maas-api-service-ca\") pod \"dcbe1377-d9f4-4038-98ba-70a1b5c6da2c\" (UID: \"dcbe1377-d9f4-4038-98ba-70a1b5c6da2c\") " Sep 16 00:53:15.466647 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:15.466622 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbe1377-d9f4-4038-98ba-70a1b5c6da2c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dcbe1377-d9f4-4038-98ba-70a1b5c6da2c" (UID: "dcbe1377-d9f4-4038-98ba-70a1b5c6da2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:53:15.567858 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:15.567815 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbe1377-d9f4-4038-98ba-70a1b5c6da2c-maas-api-service-ca\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:16.141264 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:16.141230 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:16.221348 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:16.221312 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f7c3b-c5wcs" Sep 16 00:53:16.221507 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:16.221347 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f7c3b-c5wcs" event={"ID":"dcbe1377-d9f4-4038-98ba-70a1b5c6da2c","Type":"ContainerDied","Data":"5c3d9d237faca824f2d18fc233b01ec9b813d978b55fd090f72533ebba3c8c06"} Sep 16 00:53:16.221507 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:16.221395 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c3d9d237faca824f2d18fc233b01ec9b813d978b55fd090f72533ebba3c8c06" Sep 16 00:53:16.554824 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:16.554792 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:16.873977 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:16.873875 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq"] Sep 16 00:53:16.888098 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:16.888058 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz"] Sep 16 00:53:16.954609 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:16.954570 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:22.540451 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:22.540405 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:53:23.196440 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:23.196408 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22" Sep 16 00:53:27.622238 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:27.622175 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:27.658880 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:27.658842 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x"] Sep 16 00:53:27.659125 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:27.659079 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x" podUID="8e2b07f9-744c-43d7-a97a-a6585938e0ee" containerName="payload-pre-processing" containerID="cri-o://940f565742670ec0db547139378f9e13b9f4d17b341c8a7558c752724b0ffd9c" gracePeriod=30 Sep 16 00:53:27.722050 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:27.721995 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:27.824005 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:27.823962 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:27.860567 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:27.860515 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f7c3b-c5wcs"] Sep 16 00:53:27.863240 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:27.863216 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f7c3b-c5wcs"] Sep 16 00:53:27.920956 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:27.920917 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:27.921205 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:27.921171 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x" Sep 16 00:53:27.979872 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:27.979834 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2b07f9-744c-43d7-a97a-a6585938e0ee-plugins-config-volume\") pod \"8e2b07f9-744c-43d7-a97a-a6585938e0ee\" (UID: \"8e2b07f9-744c-43d7-a97a-a6585938e0ee\") " Sep 16 00:53:27.980125 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:27.979884 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhdpm\" (UniqueName: \"kubernetes.io/projected/8e2b07f9-744c-43d7-a97a-a6585938e0ee-kube-api-access-vhdpm\") pod \"8e2b07f9-744c-43d7-a97a-a6585938e0ee\" (UID: \"8e2b07f9-744c-43d7-a97a-a6585938e0ee\") " Sep 16 00:53:27.980312 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:27.980280 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e2b07f9-744c-43d7-a97a-a6585938e0ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e2b07f9-744c-43d7-a97a-a6585938e0ee" (UID: "8e2b07f9-744c-43d7-a97a-a6585938e0ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:53:27.982320 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:27.982290 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2b07f9-744c-43d7-a97a-a6585938e0ee-kube-api-access-vhdpm" (OuterVolumeSpecName: "kube-api-access-vhdpm") pod "8e2b07f9-744c-43d7-a97a-a6585938e0ee" (UID: "8e2b07f9-744c-43d7-a97a-a6585938e0ee"). InnerVolumeSpecName "kube-api-access-vhdpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:53:28.081257 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:28.081196 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2b07f9-744c-43d7-a97a-a6585938e0ee-plugins-config-volume\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:28.081257 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:28.081249 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhdpm\" (UniqueName: \"kubernetes.io/projected/8e2b07f9-744c-43d7-a97a-a6585938e0ee-kube-api-access-vhdpm\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:28.120589 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:28.120551 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcbe1377-d9f4-4038-98ba-70a1b5c6da2c" path="/var/lib/kubelet/pods/dcbe1377-d9f4-4038-98ba-70a1b5c6da2c/volumes" Sep 16 00:53:28.296673 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:28.296579 2567 generic.go:356] "Generic (PLEG): container finished" podID="8e2b07f9-744c-43d7-a97a-a6585938e0ee" containerID="940f565742670ec0db547139378f9e13b9f4d17b341c8a7558c752724b0ffd9c" exitCode=0 Sep 16 00:53:28.296673 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:28.296625 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x" Sep 16 00:53:28.296869 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:28.296671 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x" event={"ID":"8e2b07f9-744c-43d7-a97a-a6585938e0ee","Type":"ContainerDied","Data":"940f565742670ec0db547139378f9e13b9f4d17b341c8a7558c752724b0ffd9c"} Sep 16 00:53:28.296869 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:28.296732 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x" event={"ID":"8e2b07f9-744c-43d7-a97a-a6585938e0ee","Type":"ContainerDied","Data":"6b2719a78e8f56ebafbe81f673fe6dd5f56e931104610e6027a8ecabc0661527"} Sep 16 00:53:28.296869 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:28.296757 2567 scope.go:117] "RemoveContainer" containerID="940f565742670ec0db547139378f9e13b9f4d17b341c8a7558c752724b0ffd9c" Sep 16 00:53:28.307442 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:28.307415 2567 scope.go:117] "RemoveContainer" containerID="940f565742670ec0db547139378f9e13b9f4d17b341c8a7558c752724b0ffd9c" Sep 16 00:53:28.307828 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:53:28.307808 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"940f565742670ec0db547139378f9e13b9f4d17b341c8a7558c752724b0ffd9c\": container with ID starting with 940f565742670ec0db547139378f9e13b9f4d17b341c8a7558c752724b0ffd9c not found: ID does not exist" containerID="940f565742670ec0db547139378f9e13b9f4d17b341c8a7558c752724b0ffd9c" Sep 16 00:53:28.307875 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:28.307841 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940f565742670ec0db547139378f9e13b9f4d17b341c8a7558c752724b0ffd9c"} err="failed to get container status \"940f565742670ec0db547139378f9e13b9f4d17b341c8a7558c752724b0ffd9c\": rpc error: code = NotFound desc = could not find container \"940f565742670ec0db547139378f9e13b9f4d17b341c8a7558c752724b0ffd9c\": container with ID starting with 940f565742670ec0db547139378f9e13b9f4d17b341c8a7558c752724b0ffd9c not found: ID does not exist" Sep 16 00:53:28.318308 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:28.318240 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x"] Sep 16 00:53:28.321585 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:28.321556 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0f7c3b-5f489cfcd6-72h8x"] Sep 16 00:53:28.327297 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:28.327266 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:28.414502 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:28.414462 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:29.727267 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:29.727217 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:30.120440 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:30.120403 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e2b07f9-744c-43d7-a97a-a6585938e0ee" path="/var/lib/kubelet/pods/8e2b07f9-744c-43d7-a97a-a6585938e0ee/volumes" Sep 16 00:53:30.364309 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:30.364271 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:31.231716 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:31.231675 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:31.893322 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:31.893286 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:31.897341 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:31.897295 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" podUID="bd5906f5-99bd-4608-9e84-19385f7daf27" containerName="main" containerID="cri-o://951e2f23607bc6c6ed97170c96137e9d8ae53ad763c919c6f2d0805be9bf80cc" gracePeriod=45 Sep 16 00:53:31.913944 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:31.913888 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" podUID="ddbbcc45-8ae7-4595-bc7c-257ef711cee9" containerName="main" containerID="cri-o://43f7f7cdf93f99db33c6ed76097f171c280662c225cc611c53245c8ae08d7806" gracePeriod=45 Sep 16 00:53:32.197060 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.194703 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:53:32.202898 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.202866 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:53:32.230299 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.230188 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-dshm\") pod \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " Sep 16 00:53:32.230299 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.230241 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98lrf\" (UniqueName: \"kubernetes.io/projected/bd5906f5-99bd-4608-9e84-19385f7daf27-kube-api-access-98lrf\") pod \"bd5906f5-99bd-4608-9e84-19385f7daf27\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " Sep 16 00:53:32.230299 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.230291 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-home\") pod \"bd5906f5-99bd-4608-9e84-19385f7daf27\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " Sep 16 00:53:32.230856 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.230826 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-dshm\") pod \"bd5906f5-99bd-4608-9e84-19385f7daf27\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " Sep 16 00:53:32.231016 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.230911 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-home\") pod \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " Sep 16 00:53:32.231016 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.230989 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhsvk\" (UniqueName: \"kubernetes.io/projected/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-kube-api-access-dhsvk\") pod \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " Sep 16 00:53:32.231157 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.231061 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-tmp-dir\") pod \"bd5906f5-99bd-4608-9e84-19385f7daf27\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " Sep 16 00:53:32.231157 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.231106 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-model-cache\") pod \"bd5906f5-99bd-4608-9e84-19385f7daf27\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " Sep 16 00:53:32.231157 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.231140 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-model-cache\") pod \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " Sep 16 00:53:32.231324 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.231173 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd5906f5-99bd-4608-9e84-19385f7daf27-tls-certs\") pod \"bd5906f5-99bd-4608-9e84-19385f7daf27\" (UID: \"bd5906f5-99bd-4608-9e84-19385f7daf27\") " Sep 16 00:53:32.231324 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.231230 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-tls-certs\") pod \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " Sep 16 00:53:32.231324 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.231278 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-tmp-dir\") pod \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\" (UID: \"ddbbcc45-8ae7-4595-bc7c-257ef711cee9\") " Sep 16 00:53:32.231979 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.231947 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-home" (OuterVolumeSpecName: "home") pod "bd5906f5-99bd-4608-9e84-19385f7daf27" (UID: "bd5906f5-99bd-4608-9e84-19385f7daf27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:53:32.233176 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.232110 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bd5906f5-99bd-4608-9e84-19385f7daf27" (UID: "bd5906f5-99bd-4608-9e84-19385f7daf27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:53:32.233176 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.232856 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-home" (OuterVolumeSpecName: "home") pod "ddbbcc45-8ae7-4595-bc7c-257ef711cee9" (UID: "ddbbcc45-8ae7-4595-bc7c-257ef711cee9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:53:32.233176 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.233094 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ddbbcc45-8ae7-4595-bc7c-257ef711cee9" (UID: "ddbbcc45-8ae7-4595-bc7c-257ef711cee9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:53:32.233551 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.233528 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-model-cache" (OuterVolumeSpecName: "model-cache") pod "ddbbcc45-8ae7-4595-bc7c-257ef711cee9" (UID: "ddbbcc45-8ae7-4595-bc7c-257ef711cee9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:53:32.237439 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.237389 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ddbbcc45-8ae7-4595-bc7c-257ef711cee9" (UID: "ddbbcc45-8ae7-4595-bc7c-257ef711cee9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 00:53:32.237565 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.237431 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-dshm" (OuterVolumeSpecName: "dshm") pod "ddbbcc45-8ae7-4595-bc7c-257ef711cee9" (UID: "ddbbcc45-8ae7-4595-bc7c-257ef711cee9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:53:32.237565 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.237456 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-dshm" (OuterVolumeSpecName: "dshm") pod "bd5906f5-99bd-4608-9e84-19385f7daf27" (UID: "bd5906f5-99bd-4608-9e84-19385f7daf27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:53:32.238649 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.238588 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd5906f5-99bd-4608-9e84-19385f7daf27-kube-api-access-98lrf" (OuterVolumeSpecName: "kube-api-access-98lrf") pod "bd5906f5-99bd-4608-9e84-19385f7daf27" (UID: "bd5906f5-99bd-4608-9e84-19385f7daf27"). InnerVolumeSpecName "kube-api-access-98lrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:53:32.239144 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.239052 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-model-cache" (OuterVolumeSpecName: "model-cache") pod "bd5906f5-99bd-4608-9e84-19385f7daf27" (UID: "bd5906f5-99bd-4608-9e84-19385f7daf27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:53:32.239144 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.239111 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-kube-api-access-dhsvk" (OuterVolumeSpecName: "kube-api-access-dhsvk") pod "ddbbcc45-8ae7-4595-bc7c-257ef711cee9" (UID: "ddbbcc45-8ae7-4595-bc7c-257ef711cee9"). InnerVolumeSpecName "kube-api-access-dhsvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:53:32.239557 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.239526 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd5906f5-99bd-4608-9e84-19385f7daf27-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd5906f5-99bd-4608-9e84-19385f7daf27" (UID: "bd5906f5-99bd-4608-9e84-19385f7daf27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 00:53:32.332293 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.332253 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-tmp-dir\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:32.332293 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.332283 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-model-cache\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:32.332293 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.332293 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-model-cache\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:32.332293 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.332301 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd5906f5-99bd-4608-9e84-19385f7daf27-tls-certs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:32.332640 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.332309 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-tls-certs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:32.332640 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.332316 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-tmp-dir\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:32.332640 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.332324 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-dshm\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:32.332640 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.332332 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98lrf\" (UniqueName: \"kubernetes.io/projected/bd5906f5-99bd-4608-9e84-19385f7daf27-kube-api-access-98lrf\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:32.332640 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.332340 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-home\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:32.332640 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.332348 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd5906f5-99bd-4608-9e84-19385f7daf27-dshm\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:32.332640 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.332359 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-home\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:32.332640 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.332371 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhsvk\" (UniqueName: \"kubernetes.io/projected/ddbbcc45-8ae7-4595-bc7c-257ef711cee9-kube-api-access-dhsvk\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:32.332914 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.332704 2567 generic.go:356] "Generic (PLEG): container finished" podID="ddbbcc45-8ae7-4595-bc7c-257ef711cee9" containerID="43f7f7cdf93f99db33c6ed76097f171c280662c225cc611c53245c8ae08d7806" exitCode=0 Sep 16 00:53:32.332914 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.332751 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" Sep 16 00:53:32.332914 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.332780 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" event={"ID":"ddbbcc45-8ae7-4595-bc7c-257ef711cee9","Type":"ContainerDied","Data":"43f7f7cdf93f99db33c6ed76097f171c280662c225cc611c53245c8ae08d7806"} Sep 16 00:53:32.332914 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.332814 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz" event={"ID":"ddbbcc45-8ae7-4595-bc7c-257ef711cee9","Type":"ContainerDied","Data":"ea41e9954b1124fb093e07329e044699b69272e1fa2bb0667f278ae2f3678467"} Sep 16 00:53:32.332914 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.332836 2567 scope.go:117] "RemoveContainer" containerID="43f7f7cdf93f99db33c6ed76097f171c280662c225cc611c53245c8ae08d7806" Sep 16 00:53:32.334878 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.334850 2567 generic.go:356] "Generic (PLEG): container finished" podID="bd5906f5-99bd-4608-9e84-19385f7daf27" containerID="951e2f23607bc6c6ed97170c96137e9d8ae53ad763c919c6f2d0805be9bf80cc" exitCode=0 Sep 16 00:53:32.335015 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.334918 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" Sep 16 00:53:32.335015 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.334938 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" event={"ID":"bd5906f5-99bd-4608-9e84-19385f7daf27","Type":"ContainerDied","Data":"951e2f23607bc6c6ed97170c96137e9d8ae53ad763c919c6f2d0805be9bf80cc"} Sep 16 00:53:32.335015 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.334986 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq" event={"ID":"bd5906f5-99bd-4608-9e84-19385f7daf27","Type":"ContainerDied","Data":"c3528560f0bfb0c51de6f8455ee1f3602dec7f24fad59d17ea72ae99a9d6308f"} Sep 16 00:53:32.349573 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.349537 2567 scope.go:117] "RemoveContainer" containerID="43f7f7cdf93f99db33c6ed76097f171c280662c225cc611c53245c8ae08d7806" Sep 16 00:53:32.350270 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:53:32.350242 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43f7f7cdf93f99db33c6ed76097f171c280662c225cc611c53245c8ae08d7806\": container with ID starting with 43f7f7cdf93f99db33c6ed76097f171c280662c225cc611c53245c8ae08d7806 not found: ID does not exist" containerID="43f7f7cdf93f99db33c6ed76097f171c280662c225cc611c53245c8ae08d7806" Sep 16 00:53:32.350377 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.350277 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43f7f7cdf93f99db33c6ed76097f171c280662c225cc611c53245c8ae08d7806"} err="failed to get container status \"43f7f7cdf93f99db33c6ed76097f171c280662c225cc611c53245c8ae08d7806\": rpc error: code = NotFound desc = could not find container \"43f7f7cdf93f99db33c6ed76097f171c280662c225cc611c53245c8ae08d7806\": container with ID starting with 43f7f7cdf93f99db33c6ed76097f171c280662c225cc611c53245c8ae08d7806 not found: ID does not exist" Sep 16 00:53:32.350377 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.350296 2567 scope.go:117] "RemoveContainer" containerID="951e2f23607bc6c6ed97170c96137e9d8ae53ad763c919c6f2d0805be9bf80cc" Sep 16 00:53:32.360428 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.360331 2567 scope.go:117] "RemoveContainer" containerID="951e2f23607bc6c6ed97170c96137e9d8ae53ad763c919c6f2d0805be9bf80cc" Sep 16 00:53:32.360752 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:53:32.360729 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"951e2f23607bc6c6ed97170c96137e9d8ae53ad763c919c6f2d0805be9bf80cc\": container with ID starting with 951e2f23607bc6c6ed97170c96137e9d8ae53ad763c919c6f2d0805be9bf80cc not found: ID does not exist" containerID="951e2f23607bc6c6ed97170c96137e9d8ae53ad763c919c6f2d0805be9bf80cc" Sep 16 00:53:32.360823 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.360761 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"951e2f23607bc6c6ed97170c96137e9d8ae53ad763c919c6f2d0805be9bf80cc"} err="failed to get container status \"951e2f23607bc6c6ed97170c96137e9d8ae53ad763c919c6f2d0805be9bf80cc\": rpc error: code = NotFound desc = could not find container \"951e2f23607bc6c6ed97170c96137e9d8ae53ad763c919c6f2d0805be9bf80cc\": container with ID starting with 951e2f23607bc6c6ed97170c96137e9d8ae53ad763c919c6f2d0805be9bf80cc not found: ID does not exist" Sep 16 00:53:32.363783 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.363756 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz"] Sep 16 00:53:32.366411 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.366384 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-0f7c3b/simulator-w3-0f7c3b-kserve-dc6964d7f-fjfqz"] Sep 16 00:53:32.375272 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.375238 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq"] Sep 16 00:53:32.378658 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.378632 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-0f7c3b/premium-w3-0f7c3b-kserve-ff9fbcc8b-bzxwq"] Sep 16 00:53:32.828593 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.828561 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:32.969682 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.969643 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f35fa48-cnf5n"] Sep 16 00:53:32.970132 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.970117 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd5906f5-99bd-4608-9e84-19385f7daf27" containerName="main" Sep 16 00:53:32.970184 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.970134 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd5906f5-99bd-4608-9e84-19385f7daf27" containerName="main" Sep 16 00:53:32.970184 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.970143 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddbbcc45-8ae7-4595-bc7c-257ef711cee9" containerName="main" Sep 16 00:53:32.970184 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.970148 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddbbcc45-8ae7-4595-bc7c-257ef711cee9" containerName="main" Sep 16 00:53:32.970184 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.970155 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e2b07f9-744c-43d7-a97a-a6585938e0ee" containerName="payload-pre-processing" Sep 16 00:53:32.970184 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.970161 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e2b07f9-744c-43d7-a97a-a6585938e0ee" containerName="payload-pre-processing" Sep 16 00:53:32.970184 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.970169 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcbe1377-d9f4-4038-98ba-70a1b5c6da2c" containerName="revoke-keys" Sep 16 00:53:32.970184 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.970174 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcbe1377-d9f4-4038-98ba-70a1b5c6da2c" containerName="revoke-keys" Sep 16 00:53:32.970395 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.970231 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddbbcc45-8ae7-4595-bc7c-257ef711cee9" containerName="main" Sep 16 00:53:32.970395 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.970241 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd5906f5-99bd-4608-9e84-19385f7daf27" containerName="main" Sep 16 00:53:32.970395 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.970248 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcbe1377-d9f4-4038-98ba-70a1b5c6da2c" containerName="revoke-keys" Sep 16 00:53:32.970395 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.970254 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e2b07f9-744c-43d7-a97a-a6585938e0ee" containerName="payload-pre-processing" Sep 16 00:53:32.975118 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.975094 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f35fa48-cnf5n" Sep 16 00:53:32.978784 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.978474 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-npltg\"" Sep 16 00:53:32.978784 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.978525 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 00:53:32.981156 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:32.981129 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f35fa48-cnf5n"] Sep 16 00:53:33.039000 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:33.038950 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fb7971e-eb76-43ff-9bdf-3be1bcdf250a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9f35fa48-cnf5n\" (UID: \"5fb7971e-eb76-43ff-9bdf-3be1bcdf250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f35fa48-cnf5n" Sep 16 00:53:33.139869 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:33.139759 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fb7971e-eb76-43ff-9bdf-3be1bcdf250a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9f35fa48-cnf5n\" (UID: \"5fb7971e-eb76-43ff-9bdf-3be1bcdf250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f35fa48-cnf5n" Sep 16 00:53:33.140490 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:33.140468 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fb7971e-eb76-43ff-9bdf-3be1bcdf250a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9f35fa48-cnf5n\" (UID: \"5fb7971e-eb76-43ff-9bdf-3be1bcdf250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f35fa48-cnf5n" Sep 16 00:53:33.287183 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:33.287143 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f35fa48-cnf5n" Sep 16 00:53:33.564022 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:33.563992 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f35fa48-cnf5n"] Sep 16 00:53:33.565866 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:53:33.565835 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb7971e_eb76_43ff_9bdf_3be1bcdf250a.slice/crio-4735f038deec36fdc45534ad615164a4418e1796f20569d4d80af555f730932e WatchSource:0}: Error finding container 4735f038deec36fdc45534ad615164a4418e1796f20569d4d80af555f730932e: Status 404 returned error can't find the container with id 4735f038deec36fdc45534ad615164a4418e1796f20569d4d80af555f730932e Sep 16 00:53:33.714715 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:33.714682 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:34.119747 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:34.119661 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd5906f5-99bd-4608-9e84-19385f7daf27" path="/var/lib/kubelet/pods/bd5906f5-99bd-4608-9e84-19385f7daf27/volumes" Sep 16 00:53:34.120100 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:34.120085 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddbbcc45-8ae7-4595-bc7c-257ef711cee9" path="/var/lib/kubelet/pods/ddbbcc45-8ae7-4595-bc7c-257ef711cee9/volumes" Sep 16 00:53:34.254459 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:34.254427 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:34.355164 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:34.355136 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:34.356308 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:34.356280 2567 generic.go:356] "Generic (PLEG): container finished" podID="5fb7971e-eb76-43ff-9bdf-3be1bcdf250a" containerID="05e339fa736a5f00b175d89aa02dacde3c9ebe57028cc6517d797dce9121aafd" exitCode=0 Sep 16 00:53:34.356443 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:34.356320 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f35fa48-cnf5n" event={"ID":"5fb7971e-eb76-43ff-9bdf-3be1bcdf250a","Type":"ContainerDied","Data":"05e339fa736a5f00b175d89aa02dacde3c9ebe57028cc6517d797dce9121aafd"} Sep 16 00:53:34.356443 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:34.356386 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f35fa48-cnf5n" event={"ID":"5fb7971e-eb76-43ff-9bdf-3be1bcdf250a","Type":"ContainerStarted","Data":"4735f038deec36fdc45534ad615164a4418e1796f20569d4d80af555f730932e"} Sep 16 00:53:34.527763 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:34.527665 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:34.630286 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:34.630252 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:35.503965 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:35.503934 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f35fa48-cnf5n" Sep 16 00:53:35.566764 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:35.566721 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fb7971e-eb76-43ff-9bdf-3be1bcdf250a-maas-api-service-ca\") pod \"5fb7971e-eb76-43ff-9bdf-3be1bcdf250a\" (UID: \"5fb7971e-eb76-43ff-9bdf-3be1bcdf250a\") " Sep 16 00:53:35.567195 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:35.567167 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fb7971e-eb76-43ff-9bdf-3be1bcdf250a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5fb7971e-eb76-43ff-9bdf-3be1bcdf250a" (UID: "5fb7971e-eb76-43ff-9bdf-3be1bcdf250a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:53:35.654500 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:35.654392 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:35.667730 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:35.667696 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fb7971e-eb76-43ff-9bdf-3be1bcdf250a-maas-api-service-ca\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:36.371067 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:36.371008 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f35fa48-cnf5n" Sep 16 00:53:36.371252 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:36.371074 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f35fa48-cnf5n" event={"ID":"5fb7971e-eb76-43ff-9bdf-3be1bcdf250a","Type":"ContainerDied","Data":"4735f038deec36fdc45534ad615164a4418e1796f20569d4d80af555f730932e"} Sep 16 00:53:36.371252 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:36.371140 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4735f038deec36fdc45534ad615164a4418e1796f20569d4d80af555f730932e" Sep 16 00:53:36.534692 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:36.534587 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:38.516763 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:38.516728 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:39.129891 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:39.129854 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:39.155742 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:39.155705 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22"] Sep 16 00:53:39.156126 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:39.156091 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22" podUID="20c56d99-0337-4bb5-8011-83d3b612a6d7" containerName="payload-processing" containerID="cri-o://fd99e2b5cb447bad71991a2804487fb4a503e8dc46c84612432afd14ddaf9e25" gracePeriod=30 Sep 16 00:53:39.227203 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:39.227165 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:39.327260 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:39.327215 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:39.342286 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:39.342246 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f35fa48-cnf5n"] Sep 16 00:53:39.346597 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:39.346563 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f35fa48-cnf5n"] Sep 16 00:53:39.391449 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:39.391419 2567 generic.go:356] "Generic (PLEG): container finished" podID="20c56d99-0337-4bb5-8011-83d3b612a6d7" containerID="fd99e2b5cb447bad71991a2804487fb4a503e8dc46c84612432afd14ddaf9e25" exitCode=0 Sep 16 00:53:39.391588 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:39.391490 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22" event={"ID":"20c56d99-0337-4bb5-8011-83d3b612a6d7","Type":"ContainerDied","Data":"fd99e2b5cb447bad71991a2804487fb4a503e8dc46c84612432afd14ddaf9e25"} Sep 16 00:53:39.412012 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:39.411971 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22" Sep 16 00:53:39.428366 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:39.428333 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:39.505409 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:39.505364 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dplmk\" (UniqueName: \"kubernetes.io/projected/20c56d99-0337-4bb5-8011-83d3b612a6d7-kube-api-access-dplmk\") pod \"20c56d99-0337-4bb5-8011-83d3b612a6d7\" (UID: \"20c56d99-0337-4bb5-8011-83d3b612a6d7\") " Sep 16 00:53:39.505600 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:39.505426 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c56d99-0337-4bb5-8011-83d3b612a6d7-plugins-config-volume\") pod \"20c56d99-0337-4bb5-8011-83d3b612a6d7\" (UID: \"20c56d99-0337-4bb5-8011-83d3b612a6d7\") " Sep 16 00:53:39.505864 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:39.505836 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c56d99-0337-4bb5-8011-83d3b612a6d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "20c56d99-0337-4bb5-8011-83d3b612a6d7" (UID: "20c56d99-0337-4bb5-8011-83d3b612a6d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:53:39.507741 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:39.507719 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c56d99-0337-4bb5-8011-83d3b612a6d7-kube-api-access-dplmk" (OuterVolumeSpecName: "kube-api-access-dplmk") pod "20c56d99-0337-4bb5-8011-83d3b612a6d7" (UID: "20c56d99-0337-4bb5-8011-83d3b612a6d7"). InnerVolumeSpecName "kube-api-access-dplmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:53:39.606845 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:39.606810 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dplmk\" (UniqueName: \"kubernetes.io/projected/20c56d99-0337-4bb5-8011-83d3b612a6d7-kube-api-access-dplmk\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:39.606845 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:39.606841 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c56d99-0337-4bb5-8011-83d3b612a6d7-plugins-config-volume\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:53:39.834805 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:39.834760 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:40.119168 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:40.119064 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fb7971e-eb76-43ff-9bdf-3be1bcdf250a" path="/var/lib/kubelet/pods/5fb7971e-eb76-43ff-9bdf-3be1bcdf250a/volumes" Sep 16 00:53:40.398069 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:40.397961 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22" event={"ID":"20c56d99-0337-4bb5-8011-83d3b612a6d7","Type":"ContainerDied","Data":"e7a0d3da7c58cfb746b67201543f662107b9e96a1bf14416c0c56665beb07d6a"} Sep 16 00:53:40.398069 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:40.398031 2567 scope.go:117] "RemoveContainer" containerID="fd99e2b5cb447bad71991a2804487fb4a503e8dc46c84612432afd14ddaf9e25" Sep 16 00:53:40.398314 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:40.397969 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22" Sep 16 00:53:40.419287 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:40.419253 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22"] Sep 16 00:53:40.424310 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:40.424279 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9f35fa48-86c9b47b7f-qzw22"] Sep 16 00:53:40.589423 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:40.589387 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:40.690636 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:40.690530 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:41.081571 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.081540 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:41.371563 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.371472 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99"] Sep 16 00:53:41.371909 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.371891 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fb7971e-eb76-43ff-9bdf-3be1bcdf250a" containerName="revoke-keys" Sep 16 00:53:41.371953 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.371914 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fb7971e-eb76-43ff-9bdf-3be1bcdf250a" containerName="revoke-keys" Sep 16 00:53:41.371953 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.371940 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20c56d99-0337-4bb5-8011-83d3b612a6d7" containerName="payload-processing" Sep 16 00:53:41.371953 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.371946 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="20c56d99-0337-4bb5-8011-83d3b612a6d7" containerName="payload-processing" Sep 16 00:53:41.372067 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.372010 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="20c56d99-0337-4bb5-8011-83d3b612a6d7" containerName="payload-processing" Sep 16 00:53:41.372067 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.372023 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fb7971e-eb76-43ff-9bdf-3be1bcdf250a" containerName="revoke-keys" Sep 16 00:53:41.376724 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.376692 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.379554 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.379523 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-68e143c2-openshift-default-dockercfg-2gm6z\"" Sep 16 00:53:41.384480 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.384453 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99"] Sep 16 00:53:41.397758 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.397728 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:41.497733 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.497688 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:41.526218 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.526177 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/259c4503-9088-4b01-ac37-040ff1de05c6-istio-podinfo\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.526218 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.526219 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/259c4503-9088-4b01-ac37-040ff1de05c6-istiod-ca-cert\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.526495 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.526253 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-workload-certs\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.526495 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.526342 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/259c4503-9088-4b01-ac37-040ff1de05c6-istio-token\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.526495 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.526382 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zc8q\" (UniqueName: \"kubernetes.io/projected/259c4503-9088-4b01-ac37-040ff1de05c6-kube-api-access-5zc8q\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.526495 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.526429 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-istio-data\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.526653 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.526504 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-workload-socket\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.526653 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.526526 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-istio-envoy\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.526653 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.526554 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-credential-socket\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.597317 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.597285 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:41.627526 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.627413 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zc8q\" (UniqueName: \"kubernetes.io/projected/259c4503-9088-4b01-ac37-040ff1de05c6-kube-api-access-5zc8q\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.627526 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.627481 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-istio-data\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.627802 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.627550 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-workload-socket\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.627802 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.627579 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-istio-envoy\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.627802 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.627625 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-credential-socket\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.627802 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.627717 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/259c4503-9088-4b01-ac37-040ff1de05c6-istio-podinfo\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.627802 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.627742 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/259c4503-9088-4b01-ac37-040ff1de05c6-istiod-ca-cert\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.627802 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.627777 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-workload-certs\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.628172 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.627830 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/259c4503-9088-4b01-ac37-040ff1de05c6-istio-token\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.628798 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.628762 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-istio-data\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.629032 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.629009 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-workload-socket\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.629661 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.629629 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-credential-socket\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.629930 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.629901 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-workload-certs\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.630146 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.630125 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/259c4503-9088-4b01-ac37-040ff1de05c6-istiod-ca-cert\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.633081 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.633019 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/259c4503-9088-4b01-ac37-040ff1de05c6-istio-podinfo\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.633858 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.633824 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-istio-envoy\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.637775 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.637738 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/259c4503-9088-4b01-ac37-040ff1de05c6-istio-token\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.637903 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.637872 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zc8q\" (UniqueName: \"kubernetes.io/projected/259c4503-9088-4b01-ac37-040ff1de05c6-kube-api-access-5zc8q\") pod \"e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.690125 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.690081 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:41.974077 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.974023 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99"] Sep 16 00:53:41.975906 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:53:41.975877 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259c4503_9088_4b01_ac37_040ff1de05c6.slice/crio-31169dbdbd45422ae044220df865316a0ee9e588db351e97b4b4afc60d3b7321 WatchSource:0}: Error finding container 31169dbdbd45422ae044220df865316a0ee9e588db351e97b4b4afc60d3b7321: Status 404 returned error can't find the container with id 31169dbdbd45422ae044220df865316a0ee9e588db351e97b4b4afc60d3b7321 Sep 16 00:53:41.978447 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.978410 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:53:41.978524 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.978498 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:53:41.978568 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:41.978531 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:53:42.119532 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:42.119494 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20c56d99-0337-4bb5-8011-83d3b612a6d7" path="/var/lib/kubelet/pods/20c56d99-0337-4bb5-8011-83d3b612a6d7/volumes" Sep 16 00:53:42.412698 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:42.412659 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" event={"ID":"259c4503-9088-4b01-ac37-040ff1de05c6","Type":"ContainerStarted","Data":"0e7c7448f6884da7b5e9f6c1ab9e8d1eaddb382ea4ac4f841e034cfbb15536c1"} Sep 16 00:53:42.412900 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:42.412714 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" event={"ID":"259c4503-9088-4b01-ac37-040ff1de05c6","Type":"ContainerStarted","Data":"31169dbdbd45422ae044220df865316a0ee9e588db351e97b4b4afc60d3b7321"} Sep 16 00:53:42.430866 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:42.430797 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" podStartSLOduration=1.430775781 podStartE2EDuration="1.430775781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:53:42.430082117 +0000 UTC m=+1172.792305669" watchObservedRunningTime="2026-09-16 00:53:42.430775781 +0000 UTC m=+1172.792999314" Sep 16 00:53:42.691082 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:42.690941 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:43.635618 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:43.635588 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:43.696581 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:43.696548 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:43.795742 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:43.795696 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:43.898437 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:43.898347 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:44.420020 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:44.419983 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:44.424782 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:44.424750 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:44.425912 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:44.425891 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:53:45.349276 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:45.349245 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:45.450912 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:45.450879 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:45.799625 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:45.799531 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:45.905680 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:45.905645 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:45.936449 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:45.936412 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57"] Sep 16 00:53:45.940333 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:45.940314 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57" Sep 16 00:53:45.943430 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:45.943402 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-68e143c2\"" Sep 16 00:53:45.943589 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:45.943512 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-68e143c2-dockercfg-zvb9b\"" Sep 16 00:53:45.948066 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:45.948021 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57"] Sep 16 00:53:45.966329 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:45.966290 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:46.072534 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:46.072489 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f774680-8aba-466c-b596-6a45c51c828d-plugins-config-volume\") pod \"payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57\" (UID: \"0f774680-8aba-466c-b596-6a45c51c828d\") " pod="openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57" Sep 16 00:53:46.072744 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:46.072554 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krsbm\" (UniqueName: \"kubernetes.io/projected/0f774680-8aba-466c-b596-6a45c51c828d-kube-api-access-krsbm\") pod \"payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57\" (UID: \"0f774680-8aba-466c-b596-6a45c51c828d\") " pod="openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57" Sep 16 00:53:46.173502 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:46.173444 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f774680-8aba-466c-b596-6a45c51c828d-plugins-config-volume\") pod \"payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57\" (UID: \"0f774680-8aba-466c-b596-6a45c51c828d\") " pod="openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57" Sep 16 00:53:46.173713 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:46.173531 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krsbm\" (UniqueName: \"kubernetes.io/projected/0f774680-8aba-466c-b596-6a45c51c828d-kube-api-access-krsbm\") pod \"payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57\" (UID: \"0f774680-8aba-466c-b596-6a45c51c828d\") " pod="openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57" Sep 16 00:53:46.174332 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:46.174313 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f774680-8aba-466c-b596-6a45c51c828d-plugins-config-volume\") pod \"payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57\" (UID: \"0f774680-8aba-466c-b596-6a45c51c828d\") " pod="openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57" Sep 16 00:53:46.181781 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:46.181751 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krsbm\" (UniqueName: \"kubernetes.io/projected/0f774680-8aba-466c-b596-6a45c51c828d-kube-api-access-krsbm\") pod \"payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57\" (UID: \"0f774680-8aba-466c-b596-6a45c51c828d\") " pod="openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57" Sep 16 00:53:46.253846 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:46.253804 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57" Sep 16 00:53:46.534521 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:46.534490 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57"] Sep 16 00:53:46.536553 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:53:46.536521 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f774680_8aba_466c_b596_6a45c51c828d.slice/crio-67088913788be73778e030db77ebe4f409348cb812c790f336757cb04d8a118f WatchSource:0}: Error finding container 67088913788be73778e030db77ebe4f409348cb812c790f336757cb04d8a118f: Status 404 returned error can't find the container with id 67088913788be73778e030db77ebe4f409348cb812c790f336757cb04d8a118f Sep 16 00:53:47.451258 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:47.451216 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57" event={"ID":"0f774680-8aba-466c-b596-6a45c51c828d","Type":"ContainerStarted","Data":"721973805c781bb93c9edda1c38af6a6dc08ac6df0b49a6d74932b2aa1ac39a7"} Sep 16 00:53:47.451459 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:47.451278 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57" event={"ID":"0f774680-8aba-466c-b596-6a45c51c828d","Type":"ContainerStarted","Data":"67088913788be73778e030db77ebe4f409348cb812c790f336757cb04d8a118f"} Sep 16 00:53:47.451459 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:47.451303 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57" Sep 16 00:53:47.469796 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:47.469722 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57" podStartSLOduration=2.469697508 podStartE2EDuration="2.469697508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:53:47.466601767 +0000 UTC m=+1177.828825333" watchObservedRunningTime="2026-09-16 00:53:47.469697508 +0000 UTC m=+1177.831921042" Sep 16 00:53:52.442235 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:52.442202 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:53:54.351450 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:54.351412 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:54.451622 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:54.451586 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:54.687443 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:54.687354 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:53:58.467332 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:53:58.466230 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57" Sep 16 00:54:04.557346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:04.557311 2567 generic.go:356] "Generic (PLEG): container finished" podID="259c4503-9088-4b01-ac37-040ff1de05c6" containerID="0e7c7448f6884da7b5e9f6c1ab9e8d1eaddb382ea4ac4f841e034cfbb15536c1" exitCode=0 Sep 16 00:54:04.557822 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:04.557394 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" event={"ID":"259c4503-9088-4b01-ac37-040ff1de05c6","Type":"ContainerDied","Data":"0e7c7448f6884da7b5e9f6c1ab9e8d1eaddb382ea4ac4f841e034cfbb15536c1"} Sep 16 00:54:04.557870 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:04.557840 2567 scope.go:117] "RemoveContainer" containerID="0e7c7448f6884da7b5e9f6c1ab9e8d1eaddb382ea4ac4f841e034cfbb15536c1" Sep 16 00:54:04.559140 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:04.559087 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:54:04.559202 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:04.559184 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:54:04.559240 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:04.559219 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:54:05.570932 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:05.570880 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" event={"ID":"259c4503-9088-4b01-ac37-040ff1de05c6","Type":"ContainerStarted","Data":"de2e85d733a14ff857087e22f3fc9484fcbac19d875272750d43764bce1efd9d"} Sep 16 00:54:05.690842 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:05.690799 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:54:05.691176 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:05.691153 2567 patch_prober.go:28] interesting pod/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 16 00:54:05.691233 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:05.691217 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" podUID="259c4503-9088-4b01-ac37-040ff1de05c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 16 00:54:06.691468 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:06.691430 2567 patch_prober.go:28] interesting pod/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 16 00:54:06.691896 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:06.691491 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" podUID="259c4503-9088-4b01-ac37-040ff1de05c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 16 00:54:07.448069 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:07.445111 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:07.691315 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:07.691275 2567 patch_prober.go:28] interesting pod/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 16 00:54:07.691522 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:07.691334 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" podUID="259c4503-9088-4b01-ac37-040ff1de05c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 16 00:54:08.110029 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:08.109991 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:08.691279 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:08.691234 2567 patch_prober.go:28] interesting pod/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 16 00:54:08.691506 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:08.691302 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" podUID="259c4503-9088-4b01-ac37-040ff1de05c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 16 00:54:08.757888 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:08.757825 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:08.775353 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:08.775315 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68e143c2-7k528"] Sep 16 00:54:08.777909 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:08.777889 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68e143c2-7k528" Sep 16 00:54:08.780517 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:08.780489 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-npltg\"" Sep 16 00:54:08.780668 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:08.780556 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 00:54:08.786505 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:08.786191 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68e143c2-7k528"] Sep 16 00:54:08.795400 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:08.795364 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e734f15e-0d6c-44a6-9e37-1163ab4fc5f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-68e143c2-7k528\" (UID: \"e734f15e-0d6c-44a6-9e37-1163ab4fc5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68e143c2-7k528" Sep 16 00:54:08.896591 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:08.896546 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e734f15e-0d6c-44a6-9e37-1163ab4fc5f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-68e143c2-7k528\" (UID: \"e734f15e-0d6c-44a6-9e37-1163ab4fc5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68e143c2-7k528" Sep 16 00:54:08.897254 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:08.897232 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e734f15e-0d6c-44a6-9e37-1163ab4fc5f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-68e143c2-7k528\" (UID: \"e734f15e-0d6c-44a6-9e37-1163ab4fc5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68e143c2-7k528" Sep 16 00:54:09.090834 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:09.090791 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68e143c2-7k528" Sep 16 00:54:09.383692 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:09.383662 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68e143c2-7k528"] Sep 16 00:54:09.385688 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:54:09.385661 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode734f15e_0d6c_44a6_9e37_1163ab4fc5f4.slice/crio-7cb126cdfa5ef0b2200c146aebf556ddf61c1fde6dbd5263dded9140885cd4fc WatchSource:0}: Error finding container 7cb126cdfa5ef0b2200c146aebf556ddf61c1fde6dbd5263dded9140885cd4fc: Status 404 returned error can't find the container with id 7cb126cdfa5ef0b2200c146aebf556ddf61c1fde6dbd5263dded9140885cd4fc Sep 16 00:54:09.579680 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:09.579645 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:09.597802 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:09.597762 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68e143c2-7k528" event={"ID":"e734f15e-0d6c-44a6-9e37-1163ab4fc5f4","Type":"ContainerStarted","Data":"7cb126cdfa5ef0b2200c146aebf556ddf61c1fde6dbd5263dded9140885cd4fc"} Sep 16 00:54:09.691623 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:09.691525 2567 patch_prober.go:28] interesting pod/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 16 00:54:09.691623 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:09.691594 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" podUID="259c4503-9088-4b01-ac37-040ff1de05c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 16 00:54:10.384215 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:10.384161 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:10.484300 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:10.484207 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:10.605830 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:10.605793 2567 generic.go:356] "Generic (PLEG): container finished" podID="e734f15e-0d6c-44a6-9e37-1163ab4fc5f4" containerID="dea51509032f601a4958898acd46f680633a443bc18c106e50bc4cefa8a63ebf" exitCode=0 Sep 16 00:54:10.605830 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:10.605837 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68e143c2-7k528" event={"ID":"e734f15e-0d6c-44a6-9e37-1163ab4fc5f4","Type":"ContainerDied","Data":"dea51509032f601a4958898acd46f680633a443bc18c106e50bc4cefa8a63ebf"} Sep 16 00:54:10.691127 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:10.691084 2567 patch_prober.go:28] interesting pod/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 16 00:54:10.691359 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:10.691164 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" podUID="259c4503-9088-4b01-ac37-040ff1de05c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 16 00:54:11.123335 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:11.123295 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:11.695999 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:11.695960 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:54:11.696605 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:11.696450 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:54:11.697864 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:11.697825 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:54:11.760791 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:11.760763 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68e143c2-7k528" Sep 16 00:54:11.828834 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:11.828800 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e734f15e-0d6c-44a6-9e37-1163ab4fc5f4-maas-api-service-ca\") pod \"e734f15e-0d6c-44a6-9e37-1163ab4fc5f4\" (UID: \"e734f15e-0d6c-44a6-9e37-1163ab4fc5f4\") " Sep 16 00:54:11.829257 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:11.829233 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e734f15e-0d6c-44a6-9e37-1163ab4fc5f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e734f15e-0d6c-44a6-9e37-1163ab4fc5f4" (UID: "e734f15e-0d6c-44a6-9e37-1163ab4fc5f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:54:11.842870 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:11.842838 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:11.929990 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:11.929902 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e734f15e-0d6c-44a6-9e37-1163ab4fc5f4-maas-api-service-ca\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:54:12.620115 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:12.620078 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68e143c2-7k528" Sep 16 00:54:12.620115 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:12.620091 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68e143c2-7k528" event={"ID":"e734f15e-0d6c-44a6-9e37-1163ab4fc5f4","Type":"ContainerDied","Data":"7cb126cdfa5ef0b2200c146aebf556ddf61c1fde6dbd5263dded9140885cd4fc"} Sep 16 00:54:12.620375 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:12.620182 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cb126cdfa5ef0b2200c146aebf556ddf61c1fde6dbd5263dded9140885cd4fc" Sep 16 00:54:15.073613 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:15.073582 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:15.693398 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:15.693340 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:15.723005 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:15.722968 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57"] Sep 16 00:54:15.723269 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:15.723235 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57" podUID="0f774680-8aba-466c-b596-6a45c51c828d" containerName="payload-processing" containerID="cri-o://721973805c781bb93c9edda1c38af6a6dc08ac6df0b49a6d74932b2aa1ac39a7" gracePeriod=30 Sep 16 00:54:15.792756 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:15.792728 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:15.881661 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:15.881618 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68e143c2-7k528"] Sep 16 00:54:15.882255 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:15.882215 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:15.884315 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:15.884287 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68e143c2-7k528"] Sep 16 00:54:15.938959 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:15.938913 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:15.993507 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:15.993288 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57" Sep 16 00:54:16.031069 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.030996 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99"] Sep 16 00:54:16.031288 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.031257 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" podUID="259c4503-9088-4b01-ac37-040ff1de05c6" containerName="istio-proxy" containerID="cri-o://de2e85d733a14ff857087e22f3fc9484fcbac19d875272750d43764bce1efd9d" gracePeriod=30 Sep 16 00:54:16.035526 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.035471 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:16.071919 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.071879 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krsbm\" (UniqueName: \"kubernetes.io/projected/0f774680-8aba-466c-b596-6a45c51c828d-kube-api-access-krsbm\") pod \"0f774680-8aba-466c-b596-6a45c51c828d\" (UID: \"0f774680-8aba-466c-b596-6a45c51c828d\") " Sep 16 00:54:16.072164 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.072071 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f774680-8aba-466c-b596-6a45c51c828d-plugins-config-volume\") pod \"0f774680-8aba-466c-b596-6a45c51c828d\" (UID: \"0f774680-8aba-466c-b596-6a45c51c828d\") " Sep 16 00:54:16.072348 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.072325 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f774680-8aba-466c-b596-6a45c51c828d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0f774680-8aba-466c-b596-6a45c51c828d" (UID: "0f774680-8aba-466c-b596-6a45c51c828d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:54:16.074327 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.074301 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f774680-8aba-466c-b596-6a45c51c828d-kube-api-access-krsbm" (OuterVolumeSpecName: "kube-api-access-krsbm") pod "0f774680-8aba-466c-b596-6a45c51c828d" (UID: "0f774680-8aba-466c-b596-6a45c51c828d"). InnerVolumeSpecName "kube-api-access-krsbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:54:16.127616 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.126680 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e734f15e-0d6c-44a6-9e37-1163ab4fc5f4" path="/var/lib/kubelet/pods/e734f15e-0d6c-44a6-9e37-1163ab4fc5f4/volumes" Sep 16 00:54:16.173836 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.173787 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krsbm\" (UniqueName: \"kubernetes.io/projected/0f774680-8aba-466c-b596-6a45c51c828d-kube-api-access-krsbm\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:54:16.173836 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.173830 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f774680-8aba-466c-b596-6a45c51c828d-plugins-config-volume\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:54:16.433586 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.433552 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:16.646491 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.646454 2567 generic.go:356] "Generic (PLEG): container finished" podID="0f774680-8aba-466c-b596-6a45c51c828d" containerID="721973805c781bb93c9edda1c38af6a6dc08ac6df0b49a6d74932b2aa1ac39a7" exitCode=0 Sep 16 00:54:16.646699 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.646501 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57" Sep 16 00:54:16.646699 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.646527 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57" event={"ID":"0f774680-8aba-466c-b596-6a45c51c828d","Type":"ContainerDied","Data":"721973805c781bb93c9edda1c38af6a6dc08ac6df0b49a6d74932b2aa1ac39a7"} Sep 16 00:54:16.647173 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.646907 2567 scope.go:117] "RemoveContainer" containerID="721973805c781bb93c9edda1c38af6a6dc08ac6df0b49a6d74932b2aa1ac39a7" Sep 16 00:54:16.647338 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.646869 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57" event={"ID":"0f774680-8aba-466c-b596-6a45c51c828d","Type":"ContainerDied","Data":"67088913788be73778e030db77ebe4f409348cb812c790f336757cb04d8a118f"} Sep 16 00:54:16.664935 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.664906 2567 scope.go:117] "RemoveContainer" containerID="721973805c781bb93c9edda1c38af6a6dc08ac6df0b49a6d74932b2aa1ac39a7" Sep 16 00:54:16.665356 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:54:16.665328 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"721973805c781bb93c9edda1c38af6a6dc08ac6df0b49a6d74932b2aa1ac39a7\": container with ID starting with 721973805c781bb93c9edda1c38af6a6dc08ac6df0b49a6d74932b2aa1ac39a7 not found: ID does not exist" containerID="721973805c781bb93c9edda1c38af6a6dc08ac6df0b49a6d74932b2aa1ac39a7" Sep 16 00:54:16.665484 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.665361 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"721973805c781bb93c9edda1c38af6a6dc08ac6df0b49a6d74932b2aa1ac39a7"} err="failed to get container status \"721973805c781bb93c9edda1c38af6a6dc08ac6df0b49a6d74932b2aa1ac39a7\": rpc error: code = NotFound desc = could not find container \"721973805c781bb93c9edda1c38af6a6dc08ac6df0b49a6d74932b2aa1ac39a7\": container with ID starting with 721973805c781bb93c9edda1c38af6a6dc08ac6df0b49a6d74932b2aa1ac39a7 not found: ID does not exist" Sep 16 00:54:16.672148 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.672073 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57"] Sep 16 00:54:16.675353 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.675326 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-68e143c2-d7b6bfbb5-bmx57"] Sep 16 00:54:16.798005 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.797899 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:16.896238 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.896182 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:16.996703 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:16.996658 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:17.405899 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:17.405865 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:18.057415 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:18.057373 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:18.120390 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:18.120347 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f774680-8aba-466c-b596-6a45c51c828d" path="/var/lib/kubelet/pods/0f774680-8aba-466c-b596-6a45c51c828d/volumes" Sep 16 00:54:18.481627 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:18.481521 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:18.718264 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:18.718216 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:20.742998 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:20.742958 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:54:21.281817 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.281782 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:54:21.430793 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.430737 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-workload-certs\") pod \"259c4503-9088-4b01-ac37-040ff1de05c6\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " Sep 16 00:54:21.430793 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.430797 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/259c4503-9088-4b01-ac37-040ff1de05c6-istio-token\") pod \"259c4503-9088-4b01-ac37-040ff1de05c6\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " Sep 16 00:54:21.431191 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.430831 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zc8q\" (UniqueName: \"kubernetes.io/projected/259c4503-9088-4b01-ac37-040ff1de05c6-kube-api-access-5zc8q\") pod \"259c4503-9088-4b01-ac37-040ff1de05c6\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " Sep 16 00:54:21.431191 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.430865 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/259c4503-9088-4b01-ac37-040ff1de05c6-istiod-ca-cert\") pod \"259c4503-9088-4b01-ac37-040ff1de05c6\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " Sep 16 00:54:21.431191 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.430904 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-credential-socket\") pod \"259c4503-9088-4b01-ac37-040ff1de05c6\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " Sep 16 00:54:21.431191 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.430939 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-istio-data\") pod \"259c4503-9088-4b01-ac37-040ff1de05c6\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " Sep 16 00:54:21.431191 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.430992 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/259c4503-9088-4b01-ac37-040ff1de05c6-istio-podinfo\") pod \"259c4503-9088-4b01-ac37-040ff1de05c6\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " Sep 16 00:54:21.431191 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.431018 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-workload-socket\") pod \"259c4503-9088-4b01-ac37-040ff1de05c6\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " Sep 16 00:54:21.431191 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.431081 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "259c4503-9088-4b01-ac37-040ff1de05c6" (UID: "259c4503-9088-4b01-ac37-040ff1de05c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:54:21.431191 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.431112 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-istio-envoy\") pod \"259c4503-9088-4b01-ac37-040ff1de05c6\" (UID: \"259c4503-9088-4b01-ac37-040ff1de05c6\") " Sep 16 00:54:21.431725 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.431207 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "259c4503-9088-4b01-ac37-040ff1de05c6" (UID: "259c4503-9088-4b01-ac37-040ff1de05c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:54:21.431725 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.431296 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/259c4503-9088-4b01-ac37-040ff1de05c6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "259c4503-9088-4b01-ac37-040ff1de05c6" (UID: "259c4503-9088-4b01-ac37-040ff1de05c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:54:21.431725 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.431344 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-istio-data" (OuterVolumeSpecName: "istio-data") pod "259c4503-9088-4b01-ac37-040ff1de05c6" (UID: "259c4503-9088-4b01-ac37-040ff1de05c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:54:21.431725 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.431453 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "259c4503-9088-4b01-ac37-040ff1de05c6" (UID: "259c4503-9088-4b01-ac37-040ff1de05c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:54:21.431725 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.431558 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-workload-certs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:54:21.431725 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.431577 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/259c4503-9088-4b01-ac37-040ff1de05c6-istiod-ca-cert\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:54:21.431725 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.431591 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-credential-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:54:21.431725 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.431612 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-istio-data\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:54:21.431725 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.431629 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-workload-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:54:21.433371 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.433339 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259c4503-9088-4b01-ac37-040ff1de05c6-istio-token" (OuterVolumeSpecName: "istio-token") pod "259c4503-9088-4b01-ac37-040ff1de05c6" (UID: "259c4503-9088-4b01-ac37-040ff1de05c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:54:21.433371 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.433367 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "259c4503-9088-4b01-ac37-040ff1de05c6" (UID: "259c4503-9088-4b01-ac37-040ff1de05c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:54:21.433535 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.433439 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259c4503-9088-4b01-ac37-040ff1de05c6-kube-api-access-5zc8q" (OuterVolumeSpecName: "kube-api-access-5zc8q") pod "259c4503-9088-4b01-ac37-040ff1de05c6" (UID: "259c4503-9088-4b01-ac37-040ff1de05c6"). InnerVolumeSpecName "kube-api-access-5zc8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:54:21.433535 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.433456 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/259c4503-9088-4b01-ac37-040ff1de05c6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "259c4503-9088-4b01-ac37-040ff1de05c6" (UID: "259c4503-9088-4b01-ac37-040ff1de05c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 00:54:21.532819 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.532761 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/259c4503-9088-4b01-ac37-040ff1de05c6-istio-token\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:54:21.532819 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.532813 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zc8q\" (UniqueName: \"kubernetes.io/projected/259c4503-9088-4b01-ac37-040ff1de05c6-kube-api-access-5zc8q\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:54:21.533110 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.532844 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/259c4503-9088-4b01-ac37-040ff1de05c6-istio-podinfo\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:54:21.533110 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.532858 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/259c4503-9088-4b01-ac37-040ff1de05c6-istio-envoy\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:54:21.681585 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.681482 2567 generic.go:356] "Generic (PLEG): container finished" podID="259c4503-9088-4b01-ac37-040ff1de05c6" containerID="de2e85d733a14ff857087e22f3fc9484fcbac19d875272750d43764bce1efd9d" exitCode=0 Sep 16 00:54:21.681585 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.681539 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" Sep 16 00:54:21.681585 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.681568 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" event={"ID":"259c4503-9088-4b01-ac37-040ff1de05c6","Type":"ContainerDied","Data":"de2e85d733a14ff857087e22f3fc9484fcbac19d875272750d43764bce1efd9d"} Sep 16 00:54:21.681856 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.681635 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99" event={"ID":"259c4503-9088-4b01-ac37-040ff1de05c6","Type":"ContainerDied","Data":"31169dbdbd45422ae044220df865316a0ee9e588db351e97b4b4afc60d3b7321"} Sep 16 00:54:21.681856 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.681667 2567 scope.go:117] "RemoveContainer" containerID="de2e85d733a14ff857087e22f3fc9484fcbac19d875272750d43764bce1efd9d" Sep 16 00:54:21.692542 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.692521 2567 scope.go:117] "RemoveContainer" containerID="0e7c7448f6884da7b5e9f6c1ab9e8d1eaddb382ea4ac4f841e034cfbb15536c1" Sep 16 00:54:21.708264 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.708230 2567 scope.go:117] "RemoveContainer" containerID="de2e85d733a14ff857087e22f3fc9484fcbac19d875272750d43764bce1efd9d" Sep 16 00:54:21.708723 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:54:21.708699 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2e85d733a14ff857087e22f3fc9484fcbac19d875272750d43764bce1efd9d\": container with ID starting with de2e85d733a14ff857087e22f3fc9484fcbac19d875272750d43764bce1efd9d not found: ID does not exist" containerID="de2e85d733a14ff857087e22f3fc9484fcbac19d875272750d43764bce1efd9d" Sep 16 00:54:21.708781 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.708734 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2e85d733a14ff857087e22f3fc9484fcbac19d875272750d43764bce1efd9d"} err="failed to get container status \"de2e85d733a14ff857087e22f3fc9484fcbac19d875272750d43764bce1efd9d\": rpc error: code = NotFound desc = could not find container \"de2e85d733a14ff857087e22f3fc9484fcbac19d875272750d43764bce1efd9d\": container with ID starting with de2e85d733a14ff857087e22f3fc9484fcbac19d875272750d43764bce1efd9d not found: ID does not exist" Sep 16 00:54:21.708781 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.708754 2567 scope.go:117] "RemoveContainer" containerID="0e7c7448f6884da7b5e9f6c1ab9e8d1eaddb382ea4ac4f841e034cfbb15536c1" Sep 16 00:54:21.709107 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:54:21.709079 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e7c7448f6884da7b5e9f6c1ab9e8d1eaddb382ea4ac4f841e034cfbb15536c1\": container with ID starting with 0e7c7448f6884da7b5e9f6c1ab9e8d1eaddb382ea4ac4f841e034cfbb15536c1 not found: ID does not exist" containerID="0e7c7448f6884da7b5e9f6c1ab9e8d1eaddb382ea4ac4f841e034cfbb15536c1" Sep 16 00:54:21.709197 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.709116 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e7c7448f6884da7b5e9f6c1ab9e8d1eaddb382ea4ac4f841e034cfbb15536c1"} err="failed to get container status \"0e7c7448f6884da7b5e9f6c1ab9e8d1eaddb382ea4ac4f841e034cfbb15536c1\": rpc error: code = NotFound desc = could not find container \"0e7c7448f6884da7b5e9f6c1ab9e8d1eaddb382ea4ac4f841e034cfbb15536c1\": container with ID starting with 0e7c7448f6884da7b5e9f6c1ab9e8d1eaddb382ea4ac4f841e034cfbb15536c1 not found: ID does not exist" Sep 16 00:54:21.709321 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.709299 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99"] Sep 16 00:54:21.716658 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:21.716622 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-68e143c2-openshift-default-5dccfb7568-82t99"] Sep 16 00:54:22.122565 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:22.122519 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="259c4503-9088-4b01-ac37-040ff1de05c6" path="/var/lib/kubelet/pods/259c4503-9088-4b01-ac37-040ff1de05c6/volumes" Sep 16 00:54:24.426482 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:24.426454 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:24.536848 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:24.536814 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:24.626734 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:24.626685 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:24.727149 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:24.727016 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:25.184978 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:25.184928 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:31.600939 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.600898 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw"] Sep 16 00:54:31.601454 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.601312 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="259c4503-9088-4b01-ac37-040ff1de05c6" containerName="istio-proxy" Sep 16 00:54:31.601454 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.601325 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="259c4503-9088-4b01-ac37-040ff1de05c6" containerName="istio-proxy" Sep 16 00:54:31.601454 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.601335 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f774680-8aba-466c-b596-6a45c51c828d" containerName="payload-processing" Sep 16 00:54:31.601454 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.601340 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f774680-8aba-466c-b596-6a45c51c828d" containerName="payload-processing" Sep 16 00:54:31.601454 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.601349 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="259c4503-9088-4b01-ac37-040ff1de05c6" containerName="istio-proxy" Sep 16 00:54:31.601454 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.601355 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="259c4503-9088-4b01-ac37-040ff1de05c6" containerName="istio-proxy" Sep 16 00:54:31.601454 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.601365 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e734f15e-0d6c-44a6-9e37-1163ab4fc5f4" containerName="revoke-keys" Sep 16 00:54:31.601454 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.601371 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e734f15e-0d6c-44a6-9e37-1163ab4fc5f4" containerName="revoke-keys" Sep 16 00:54:31.601454 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.601436 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="259c4503-9088-4b01-ac37-040ff1de05c6" containerName="istio-proxy" Sep 16 00:54:31.601454 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.601446 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f774680-8aba-466c-b596-6a45c51c828d" containerName="payload-processing" Sep 16 00:54:31.601454 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.601453 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="259c4503-9088-4b01-ac37-040ff1de05c6" containerName="istio-proxy" Sep 16 00:54:31.601454 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.601460 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e734f15e-0d6c-44a6-9e37-1163ab4fc5f4" containerName="revoke-keys" Sep 16 00:54:31.606446 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.606424 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:31.610350 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.610318 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-2fa86d\"/\"auth-test-model-2fa86d-kserve-self-signed-certs\"" Sep 16 00:54:31.610350 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.610339 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-2fa86d\"/\"openshift-service-ca.crt\"" Sep 16 00:54:31.610589 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.610318 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-2fa86d\"/\"default-dockercfg-rfc58\"" Sep 16 00:54:31.610589 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.610332 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-2fa86d\"/\"kube-root-ca.crt\"" Sep 16 00:54:31.614466 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.614433 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw"] Sep 16 00:54:31.690386 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.690351 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:31.735816 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.735778 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nshl\" (UniqueName: \"kubernetes.io/projected/c294a8d7-a9f7-448c-8c69-495698d7da1c-kube-api-access-7nshl\") pod \"auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:31.736011 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.735830 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c294a8d7-a9f7-448c-8c69-495698d7da1c-tls-certs\") pod \"auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:31.736011 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.735920 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-tmp-dir\") pod \"auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:31.736011 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.735960 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-dshm\") pod \"auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:31.736011 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.735989 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-model-cache\") pod \"auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:31.736170 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.736033 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-home\") pod \"auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:31.837358 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.837315 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-tmp-dir\") pod \"auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:31.837358 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.837365 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-dshm\") pod \"auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:31.837637 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.837493 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-model-cache\") pod \"auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:31.837637 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.837559 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-home\") pod \"auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:31.837748 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.837654 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nshl\" (UniqueName: \"kubernetes.io/projected/c294a8d7-a9f7-448c-8c69-495698d7da1c-kube-api-access-7nshl\") pod \"auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:31.837748 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.837706 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c294a8d7-a9f7-448c-8c69-495698d7da1c-tls-certs\") pod \"auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:31.837748 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.837709 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-tmp-dir\") pod \"auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:31.837884 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.837826 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-model-cache\") pod \"auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:31.837884 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.837872 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-home\") pod \"auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:31.839903 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.839873 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-dshm\") pod \"auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:31.840224 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.840206 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c294a8d7-a9f7-448c-8c69-495698d7da1c-tls-certs\") pod \"auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:31.847014 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.846980 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nshl\" (UniqueName: \"kubernetes.io/projected/c294a8d7-a9f7-448c-8c69-495698d7da1c-kube-api-access-7nshl\") pod \"auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:31.918600 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:31.918499 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:32.209796 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:32.209765 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw"] Sep 16 00:54:32.211935 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:54:32.211896 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc294a8d7_a9f7_448c_8c69_495698d7da1c.slice/crio-8e5ca492ed1033874a2469fc4c4eb72b27094e851c8c10eb32e527b212939c27 WatchSource:0}: Error finding container 8e5ca492ed1033874a2469fc4c4eb72b27094e851c8c10eb32e527b212939c27: Status 404 returned error can't find the container with id 8e5ca492ed1033874a2469fc4c4eb72b27094e851c8c10eb32e527b212939c27 Sep 16 00:54:32.760917 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:32.760864 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" event={"ID":"c294a8d7-a9f7-448c-8c69-495698d7da1c","Type":"ContainerStarted","Data":"4e9de3aa5df7e40a94a698bc53514d92d1eb791307c9d3be5263875d81483b31"} Sep 16 00:54:32.761392 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:32.760924 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" event={"ID":"c294a8d7-a9f7-448c-8c69-495698d7da1c","Type":"ContainerStarted","Data":"8e5ca492ed1033874a2469fc4c4eb72b27094e851c8c10eb32e527b212939c27"} Sep 16 00:54:32.778082 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:32.777991 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" podStartSLOduration=1.77796964 podStartE2EDuration="1.77796964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:54:32.776110169 +0000 UTC m=+1223.138333703" watchObservedRunningTime="2026-09-16 00:54:32.77796964 +0000 UTC m=+1223.140193234" Sep 16 00:54:41.241788 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:41.241747 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:54:41.919437 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:41.919382 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:41.919644 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:41.919470 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:41.932464 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:41.932431 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:42.364973 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:42.364934 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:42.833815 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:42.833781 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:54:51.226666 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:51.226619 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:57.069726 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:57.069691 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:57.127320 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:57.127283 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:57.204490 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:57.204448 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:54:57.302677 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:54:57.302627 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:00.635505 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:00.635468 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:55:07.356733 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:07.356652 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:07.485326 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:07.485290 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5270e2-phhsh"] Sep 16 00:55:07.488871 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:07.488851 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5270e2-phhsh" Sep 16 00:55:07.491942 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:07.491885 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 00:55:07.492357 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:07.491956 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-npltg\"" Sep 16 00:55:07.496986 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:07.496952 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5270e2-phhsh"] Sep 16 00:55:07.504274 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:07.504240 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/729d28ef-b0de-4822-87f5-e53622ef1ff0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-5270e2-phhsh\" (UID: \"729d28ef-b0de-4822-87f5-e53622ef1ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5270e2-phhsh" Sep 16 00:55:07.604912 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:07.604866 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/729d28ef-b0de-4822-87f5-e53622ef1ff0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-5270e2-phhsh\" (UID: \"729d28ef-b0de-4822-87f5-e53622ef1ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5270e2-phhsh" Sep 16 00:55:07.605630 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:07.605604 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/729d28ef-b0de-4822-87f5-e53622ef1ff0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-5270e2-phhsh\" (UID: \"729d28ef-b0de-4822-87f5-e53622ef1ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5270e2-phhsh" Sep 16 00:55:07.802094 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:07.802060 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5270e2-phhsh" Sep 16 00:55:07.835419 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:07.835377 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:08.078428 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:08.078398 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5270e2-phhsh"] Sep 16 00:55:08.080433 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:55:08.080396 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729d28ef_b0de_4822_87f5_e53622ef1ff0.slice/crio-bbebb5a1bc732462e3ac850c630b24fe279c58f464d341ea68d0c4eff4f0fa1c WatchSource:0}: Error finding container bbebb5a1bc732462e3ac850c630b24fe279c58f464d341ea68d0c4eff4f0fa1c: Status 404 returned error can't find the container with id bbebb5a1bc732462e3ac850c630b24fe279c58f464d341ea68d0c4eff4f0fa1c Sep 16 00:55:08.720112 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:08.719989 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:08.979024 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:08.978927 2567 generic.go:356] "Generic (PLEG): container finished" podID="729d28ef-b0de-4822-87f5-e53622ef1ff0" containerID="af4c7eb83f993ac7e758b9238d33909eb33dc0d2582ecb5c56b6a58ce239d70d" exitCode=0 Sep 16 00:55:08.979024 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:08.979000 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5270e2-phhsh" event={"ID":"729d28ef-b0de-4822-87f5-e53622ef1ff0","Type":"ContainerDied","Data":"af4c7eb83f993ac7e758b9238d33909eb33dc0d2582ecb5c56b6a58ce239d70d"} Sep 16 00:55:08.979254 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:08.979069 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5270e2-phhsh" event={"ID":"729d28ef-b0de-4822-87f5-e53622ef1ff0","Type":"ContainerStarted","Data":"bbebb5a1bc732462e3ac850c630b24fe279c58f464d341ea68d0c4eff4f0fa1c"} Sep 16 00:55:09.534058 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:09.534008 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:10.131219 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:10.131188 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5270e2-phhsh" Sep 16 00:55:10.217617 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:10.217584 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:10.227611 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:10.227579 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/729d28ef-b0de-4822-87f5-e53622ef1ff0-maas-api-service-ca\") pod \"729d28ef-b0de-4822-87f5-e53622ef1ff0\" (UID: \"729d28ef-b0de-4822-87f5-e53622ef1ff0\") " Sep 16 00:55:10.227950 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:10.227924 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729d28ef-b0de-4822-87f5-e53622ef1ff0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "729d28ef-b0de-4822-87f5-e53622ef1ff0" (UID: "729d28ef-b0de-4822-87f5-e53622ef1ff0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:55:10.228082 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:10.228007 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/729d28ef-b0de-4822-87f5-e53622ef1ff0-maas-api-service-ca\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:55:10.992590 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:10.992563 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5270e2-phhsh" Sep 16 00:55:10.992590 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:10.992573 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5270e2-phhsh" event={"ID":"729d28ef-b0de-4822-87f5-e53622ef1ff0","Type":"ContainerDied","Data":"bbebb5a1bc732462e3ac850c630b24fe279c58f464d341ea68d0c4eff4f0fa1c"} Sep 16 00:55:10.992839 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:10.992620 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbebb5a1bc732462e3ac850c630b24fe279c58f464d341ea68d0c4eff4f0fa1c" Sep 16 00:55:12.346322 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:12.346281 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv"] Sep 16 00:55:12.424720 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:12.424679 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:13.204307 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:13.204268 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:19.055070 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:19.055009 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:19.132804 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:19.132772 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:19.241346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:19.241317 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:19.332354 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:19.332252 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:23.188422 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:23.188385 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:23.252507 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:23.252465 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:23.310726 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:23.310689 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:23.385196 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:23.385153 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5270e2-phhsh"] Sep 16 00:55:23.387211 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:23.387156 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:23.390090 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:23.390058 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5270e2-phhsh"] Sep 16 00:55:23.443627 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:23.443521 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:23.790295 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:23.790199 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:23.968815 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:23.968775 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:24.119380 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:24.119338 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="729d28ef-b0de-4822-87f5-e53622ef1ff0" path="/var/lib/kubelet/pods/729d28ef-b0de-4822-87f5-e53622ef1ff0/volumes" Sep 16 00:55:24.145227 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:24.145191 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:55:24.803630 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:24.803554 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:27.367815 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:27.367768 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" podUID="c164b60c-870c-42bc-be7f-59f517de59f2" containerName="main" containerID="cri-o://2648d37329e185d9a5b02637c0ad439e277c260b448755b0df79272e2fd81b7e" gracePeriod=45 Sep 16 00:55:27.631852 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:27.631778 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:55:27.704855 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:27.704820 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c164b60c-870c-42bc-be7f-59f517de59f2-tls-certs\") pod \"c164b60c-870c-42bc-be7f-59f517de59f2\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " Sep 16 00:55:27.705165 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:27.704879 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-home\") pod \"c164b60c-870c-42bc-be7f-59f517de59f2\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " Sep 16 00:55:27.705165 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:27.704920 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-dshm\") pod \"c164b60c-870c-42bc-be7f-59f517de59f2\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " Sep 16 00:55:27.705165 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:27.704971 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-model-cache\") pod \"c164b60c-870c-42bc-be7f-59f517de59f2\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " Sep 16 00:55:27.705165 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:27.705027 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ltcx\" (UniqueName: \"kubernetes.io/projected/c164b60c-870c-42bc-be7f-59f517de59f2-kube-api-access-6ltcx\") pod \"c164b60c-870c-42bc-be7f-59f517de59f2\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " Sep 16 00:55:27.705165 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:27.705093 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-tmp-dir\") pod \"c164b60c-870c-42bc-be7f-59f517de59f2\" (UID: \"c164b60c-870c-42bc-be7f-59f517de59f2\") " Sep 16 00:55:27.705464 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:27.705341 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-home" (OuterVolumeSpecName: "home") pod "c164b60c-870c-42bc-be7f-59f517de59f2" (UID: "c164b60c-870c-42bc-be7f-59f517de59f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:55:27.705464 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:27.705354 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "c164b60c-870c-42bc-be7f-59f517de59f2" (UID: "c164b60c-870c-42bc-be7f-59f517de59f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:55:27.705573 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:27.705554 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c164b60c-870c-42bc-be7f-59f517de59f2" (UID: "c164b60c-870c-42bc-be7f-59f517de59f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:55:27.707630 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:27.707603 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c164b60c-870c-42bc-be7f-59f517de59f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c164b60c-870c-42bc-be7f-59f517de59f2" (UID: "c164b60c-870c-42bc-be7f-59f517de59f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 00:55:27.707743 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:27.707719 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c164b60c-870c-42bc-be7f-59f517de59f2-kube-api-access-6ltcx" (OuterVolumeSpecName: "kube-api-access-6ltcx") pod "c164b60c-870c-42bc-be7f-59f517de59f2" (UID: "c164b60c-870c-42bc-be7f-59f517de59f2"). InnerVolumeSpecName "kube-api-access-6ltcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:55:27.707804 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:27.707793 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-dshm" (OuterVolumeSpecName: "dshm") pod "c164b60c-870c-42bc-be7f-59f517de59f2" (UID: "c164b60c-870c-42bc-be7f-59f517de59f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:55:27.806380 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:27.806338 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ltcx\" (UniqueName: \"kubernetes.io/projected/c164b60c-870c-42bc-be7f-59f517de59f2-kube-api-access-6ltcx\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:55:27.806380 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:27.806378 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-tmp-dir\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:55:27.806380 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:27.806391 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c164b60c-870c-42bc-be7f-59f517de59f2-tls-certs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:55:27.806637 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:27.806400 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-home\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:55:27.806637 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:27.806408 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-dshm\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:55:27.806637 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:27.806415 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c164b60c-870c-42bc-be7f-59f517de59f2-model-cache\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:55:28.099725 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:28.099689 2567 generic.go:356] "Generic (PLEG): container finished" podID="c164b60c-870c-42bc-be7f-59f517de59f2" containerID="2648d37329e185d9a5b02637c0ad439e277c260b448755b0df79272e2fd81b7e" exitCode=0 Sep 16 00:55:28.099890 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:28.099748 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" event={"ID":"c164b60c-870c-42bc-be7f-59f517de59f2","Type":"ContainerDied","Data":"2648d37329e185d9a5b02637c0ad439e277c260b448755b0df79272e2fd81b7e"} Sep 16 00:55:28.099890 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:28.099757 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" Sep 16 00:55:28.099890 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:28.099783 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv" event={"ID":"c164b60c-870c-42bc-be7f-59f517de59f2","Type":"ContainerDied","Data":"7b2e86b9f8d8c0d093c62a03ffddc0a017d7790f292c4721bce9b506500a85c2"} Sep 16 00:55:28.099890 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:28.099800 2567 scope.go:117] "RemoveContainer" containerID="2648d37329e185d9a5b02637c0ad439e277c260b448755b0df79272e2fd81b7e" Sep 16 00:55:28.110350 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:28.110329 2567 scope.go:117] "RemoveContainer" containerID="2648d37329e185d9a5b02637c0ad439e277c260b448755b0df79272e2fd81b7e" Sep 16 00:55:28.110680 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:55:28.110662 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2648d37329e185d9a5b02637c0ad439e277c260b448755b0df79272e2fd81b7e\": container with ID starting with 2648d37329e185d9a5b02637c0ad439e277c260b448755b0df79272e2fd81b7e not found: ID does not exist" containerID="2648d37329e185d9a5b02637c0ad439e277c260b448755b0df79272e2fd81b7e" Sep 16 00:55:28.110721 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:28.110691 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2648d37329e185d9a5b02637c0ad439e277c260b448755b0df79272e2fd81b7e"} err="failed to get container status \"2648d37329e185d9a5b02637c0ad439e277c260b448755b0df79272e2fd81b7e\": rpc error: code = NotFound desc = could not find container \"2648d37329e185d9a5b02637c0ad439e277c260b448755b0df79272e2fd81b7e\": container with ID starting with 2648d37329e185d9a5b02637c0ad439e277c260b448755b0df79272e2fd81b7e not found: ID does not exist" Sep 16 00:55:28.121921 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:28.121888 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv"] Sep 16 00:55:28.126003 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:28.125967 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5270e2/distinct-w0-5270e2-kserve-5cdc6db996-62kjv"] Sep 16 00:55:29.453646 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:29.453612 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:29.549173 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:29.549118 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:29.728382 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:29.728278 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn"] Sep 16 00:55:29.728617 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:29.728584 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" podUID="3eff9577-aa6f-4e10-8538-ba0d09f132eb" containerName="istio-proxy" containerID="cri-o://32a408ba2e0fe12373d9cdb74bbe42e7fc8a1a7b119c421186e0d24d7a6d068f" gracePeriod=30 Sep 16 00:55:29.729388 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:29.729357 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:29.828831 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:29.828792 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:30.119312 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:30.119277 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c164b60c-870c-42bc-be7f-59f517de59f2" path="/var/lib/kubelet/pods/c164b60c-870c-42bc-be7f-59f517de59f2/volumes" Sep 16 00:55:30.624722 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:30.624685 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:33.431764 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:33.431721 2567 patch_prober.go:28] interesting pod/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 00:55:33.432204 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:33.431790 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" podUID="3eff9577-aa6f-4e10-8538-ba0d09f132eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 00:55:34.997785 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:34.997711 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:55:35.076637 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.076600 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-data\") pod \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " Sep 16 00:55:35.076637 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.076653 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-workload-certs\") pod \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " Sep 16 00:55:35.076905 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.076685 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-workload-socket\") pod \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " Sep 16 00:55:35.076905 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.076706 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istiod-ca-cert\") pod \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " Sep 16 00:55:35.076905 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.076731 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-credential-socket\") pod \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " Sep 16 00:55:35.076905 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.076747 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-envoy\") pod \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " Sep 16 00:55:35.076905 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.076787 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd7jn\" (UniqueName: \"kubernetes.io/projected/3eff9577-aa6f-4e10-8538-ba0d09f132eb-kube-api-access-sd7jn\") pod \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " Sep 16 00:55:35.076905 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.076811 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-token\") pod \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " Sep 16 00:55:35.077256 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.076908 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-podinfo\") pod \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\" (UID: \"3eff9577-aa6f-4e10-8538-ba0d09f132eb\") " Sep 16 00:55:35.077256 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.076945 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-data" (OuterVolumeSpecName: "istio-data") pod "3eff9577-aa6f-4e10-8538-ba0d09f132eb" (UID: "3eff9577-aa6f-4e10-8538-ba0d09f132eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:55:35.077256 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.076958 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3eff9577-aa6f-4e10-8538-ba0d09f132eb" (UID: "3eff9577-aa6f-4e10-8538-ba0d09f132eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:55:35.077256 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.077070 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3eff9577-aa6f-4e10-8538-ba0d09f132eb" (UID: "3eff9577-aa6f-4e10-8538-ba0d09f132eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:55:35.077256 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.077102 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3eff9577-aa6f-4e10-8538-ba0d09f132eb" (UID: "3eff9577-aa6f-4e10-8538-ba0d09f132eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:55:35.077256 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.077199 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3eff9577-aa6f-4e10-8538-ba0d09f132eb" (UID: "3eff9577-aa6f-4e10-8538-ba0d09f132eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:55:35.077618 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.077332 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-workload-certs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:55:35.077618 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.077348 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-workload-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:55:35.077618 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.077357 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istiod-ca-cert\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:55:35.077618 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.077365 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-credential-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:55:35.077618 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.077373 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-data\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:55:35.079536 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.079499 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3eff9577-aa6f-4e10-8538-ba0d09f132eb" (UID: "3eff9577-aa6f-4e10-8538-ba0d09f132eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 00:55:35.079536 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.079511 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-token" (OuterVolumeSpecName: "istio-token") pod "3eff9577-aa6f-4e10-8538-ba0d09f132eb" (UID: "3eff9577-aa6f-4e10-8538-ba0d09f132eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:55:35.079740 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.079722 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3eff9577-aa6f-4e10-8538-ba0d09f132eb" (UID: "3eff9577-aa6f-4e10-8538-ba0d09f132eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:55:35.079826 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.079810 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eff9577-aa6f-4e10-8538-ba0d09f132eb-kube-api-access-sd7jn" (OuterVolumeSpecName: "kube-api-access-sd7jn") pod "3eff9577-aa6f-4e10-8538-ba0d09f132eb" (UID: "3eff9577-aa6f-4e10-8538-ba0d09f132eb"). InnerVolumeSpecName "kube-api-access-sd7jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:55:35.145862 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.145830 2567 generic.go:356] "Generic (PLEG): container finished" podID="3eff9577-aa6f-4e10-8538-ba0d09f132eb" containerID="32a408ba2e0fe12373d9cdb74bbe42e7fc8a1a7b119c421186e0d24d7a6d068f" exitCode=0 Sep 16 00:55:35.146064 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.145880 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" Sep 16 00:55:35.146064 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.145914 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" event={"ID":"3eff9577-aa6f-4e10-8538-ba0d09f132eb","Type":"ContainerDied","Data":"32a408ba2e0fe12373d9cdb74bbe42e7fc8a1a7b119c421186e0d24d7a6d068f"} Sep 16 00:55:35.146064 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.145975 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn" event={"ID":"3eff9577-aa6f-4e10-8538-ba0d09f132eb","Type":"ContainerDied","Data":"bd4acee0519959d7e632847db9ba9451c5c2d917623ae63e2b07e35bd12b25ef"} Sep 16 00:55:35.146064 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.146006 2567 scope.go:117] "RemoveContainer" containerID="32a408ba2e0fe12373d9cdb74bbe42e7fc8a1a7b119c421186e0d24d7a6d068f" Sep 16 00:55:35.156716 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.156689 2567 scope.go:117] "RemoveContainer" containerID="ff74aa81986dfeb5bb051c9a0cf24d8dc0f4a116526a28b9e1d4481998ca664c" Sep 16 00:55:35.166943 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.166916 2567 scope.go:117] "RemoveContainer" containerID="32a408ba2e0fe12373d9cdb74bbe42e7fc8a1a7b119c421186e0d24d7a6d068f" Sep 16 00:55:35.167400 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:55:35.167378 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a408ba2e0fe12373d9cdb74bbe42e7fc8a1a7b119c421186e0d24d7a6d068f\": container with ID starting with 32a408ba2e0fe12373d9cdb74bbe42e7fc8a1a7b119c421186e0d24d7a6d068f not found: ID does not exist" containerID="32a408ba2e0fe12373d9cdb74bbe42e7fc8a1a7b119c421186e0d24d7a6d068f" Sep 16 00:55:35.167471 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.167412 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a408ba2e0fe12373d9cdb74bbe42e7fc8a1a7b119c421186e0d24d7a6d068f"} err="failed to get container status \"32a408ba2e0fe12373d9cdb74bbe42e7fc8a1a7b119c421186e0d24d7a6d068f\": rpc error: code = NotFound desc = could not find container \"32a408ba2e0fe12373d9cdb74bbe42e7fc8a1a7b119c421186e0d24d7a6d068f\": container with ID starting with 32a408ba2e0fe12373d9cdb74bbe42e7fc8a1a7b119c421186e0d24d7a6d068f not found: ID does not exist" Sep 16 00:55:35.167471 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.167433 2567 scope.go:117] "RemoveContainer" containerID="ff74aa81986dfeb5bb051c9a0cf24d8dc0f4a116526a28b9e1d4481998ca664c" Sep 16 00:55:35.167770 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:55:35.167748 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff74aa81986dfeb5bb051c9a0cf24d8dc0f4a116526a28b9e1d4481998ca664c\": container with ID starting with ff74aa81986dfeb5bb051c9a0cf24d8dc0f4a116526a28b9e1d4481998ca664c not found: ID does not exist" containerID="ff74aa81986dfeb5bb051c9a0cf24d8dc0f4a116526a28b9e1d4481998ca664c" Sep 16 00:55:35.167833 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.167780 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff74aa81986dfeb5bb051c9a0cf24d8dc0f4a116526a28b9e1d4481998ca664c"} err="failed to get container status \"ff74aa81986dfeb5bb051c9a0cf24d8dc0f4a116526a28b9e1d4481998ca664c\": rpc error: code = NotFound desc = could not find container \"ff74aa81986dfeb5bb051c9a0cf24d8dc0f4a116526a28b9e1d4481998ca664c\": container with ID starting with ff74aa81986dfeb5bb051c9a0cf24d8dc0f4a116526a28b9e1d4481998ca664c not found: ID does not exist" Sep 16 00:55:35.169121 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.169100 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn"] Sep 16 00:55:35.173740 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.173711 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5270e2-openshift-default-5f679b97d6-6vsfn"] Sep 16 00:55:35.178161 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.178139 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd7jn\" (UniqueName: \"kubernetes.io/projected/3eff9577-aa6f-4e10-8538-ba0d09f132eb-kube-api-access-sd7jn\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:55:35.178245 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.178164 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-token\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:55:35.178245 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.178176 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-podinfo\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:55:35.178245 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:35.178189 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eff9577-aa6f-4e10-8538-ba0d09f132eb-istio-envoy\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:55:36.119178 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:36.119136 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eff9577-aa6f-4e10-8538-ba0d09f132eb" path="/var/lib/kubelet/pods/3eff9577-aa6f-4e10-8538-ba0d09f132eb/volumes" Sep 16 00:55:36.371402 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:36.371313 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:36.436292 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:36.436252 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:36.520276 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:36.520219 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:36.617421 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:36.617382 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:36.717890 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:36.717802 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:37.334888 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:37.334833 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:37.433325 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:37.433291 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:37.533508 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:37.533476 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:39.100425 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:39.100385 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:44.144460 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:44.144417 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:55:44.734887 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:44.734851 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:44.834885 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:44.834845 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:45.422320 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:45.422272 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:49.248843 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:49.248792 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:49.299882 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:49.299852 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:49.622406 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:49.622366 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:51.029412 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:51.029374 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:51.119392 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:51.119359 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:51.264241 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:51.264203 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:51.581755 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:51.581724 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:52.117438 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:52.117403 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:54.025734 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:54.025699 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:54.695375 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:54.695340 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:54.753142 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:54.753106 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:55.023410 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:55.023318 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:55.643435 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:55.643399 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:55.744313 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:55.744278 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:55.844393 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:55.844364 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:56.639975 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:56.639933 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:56.736011 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:56.735979 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:56.872812 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:56.872780 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:57.091728 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:57.091699 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:55:57.193504 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:55:57.193472 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:00.653772 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:00.653733 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:56:15.546313 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:15.546270 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:56:18.825184 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:18.825137 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:19.488816 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:19.488776 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:19.670737 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:19.670704 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:20.370923 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:20.370885 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:20.473390 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:20.473349 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:20.836274 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:20.836238 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:22.667884 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:22.667851 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:26.885353 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:26.885321 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:26.983861 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:26.983824 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:27.269950 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.269846 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-228a5f03-2v9vh"] Sep 16 00:56:27.270260 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.270246 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eff9577-aa6f-4e10-8538-ba0d09f132eb" containerName="istio-proxy" Sep 16 00:56:27.270312 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.270263 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eff9577-aa6f-4e10-8538-ba0d09f132eb" containerName="istio-proxy" Sep 16 00:56:27.270312 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.270271 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="729d28ef-b0de-4822-87f5-e53622ef1ff0" containerName="revoke-keys" Sep 16 00:56:27.270312 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.270276 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="729d28ef-b0de-4822-87f5-e53622ef1ff0" containerName="revoke-keys" Sep 16 00:56:27.270312 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.270289 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eff9577-aa6f-4e10-8538-ba0d09f132eb" containerName="istio-proxy" Sep 16 00:56:27.270312 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.270295 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eff9577-aa6f-4e10-8538-ba0d09f132eb" containerName="istio-proxy" Sep 16 00:56:27.270312 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.270313 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c164b60c-870c-42bc-be7f-59f517de59f2" containerName="main" Sep 16 00:56:27.270496 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.270318 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c164b60c-870c-42bc-be7f-59f517de59f2" containerName="main" Sep 16 00:56:27.270496 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.270379 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eff9577-aa6f-4e10-8538-ba0d09f132eb" containerName="istio-proxy" Sep 16 00:56:27.270496 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.270386 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c164b60c-870c-42bc-be7f-59f517de59f2" containerName="main" Sep 16 00:56:27.270496 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.270395 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="729d28ef-b0de-4822-87f5-e53622ef1ff0" containerName="revoke-keys" Sep 16 00:56:27.275470 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.275423 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-228a5f03-2v9vh" Sep 16 00:56:27.278333 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.278300 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-npltg\"" Sep 16 00:56:27.278517 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.278300 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 00:56:27.280457 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.280380 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-228a5f03-2v9vh"] Sep 16 00:56:27.386557 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.386508 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e1b23e-bbdc-4253-9625-90b8a3315ef9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-228a5f03-2v9vh\" (UID: \"f1e1b23e-bbdc-4253-9625-90b8a3315ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-228a5f03-2v9vh" Sep 16 00:56:27.487641 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.487606 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e1b23e-bbdc-4253-9625-90b8a3315ef9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-228a5f03-2v9vh\" (UID: \"f1e1b23e-bbdc-4253-9625-90b8a3315ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-228a5f03-2v9vh" Sep 16 00:56:27.488297 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.488275 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e1b23e-bbdc-4253-9625-90b8a3315ef9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-228a5f03-2v9vh\" (UID: \"f1e1b23e-bbdc-4253-9625-90b8a3315ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-228a5f03-2v9vh" Sep 16 00:56:27.588650 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.588610 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-228a5f03-2v9vh" Sep 16 00:56:27.869459 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:27.869425 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-228a5f03-2v9vh"] Sep 16 00:56:27.871188 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:56:27.871153 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e1b23e_bbdc_4253_9625_90b8a3315ef9.slice/crio-ac536f34f784c34deca43b31b6086b76131f8870204eacef2659c51e5df58209 WatchSource:0}: Error finding container ac536f34f784c34deca43b31b6086b76131f8870204eacef2659c51e5df58209: Status 404 returned error can't find the container with id ac536f34f784c34deca43b31b6086b76131f8870204eacef2659c51e5df58209 Sep 16 00:56:28.047129 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:28.047095 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:28.466078 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:28.465947 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-228a5f03-2v9vh" event={"ID":"f1e1b23e-bbdc-4253-9625-90b8a3315ef9","Type":"ContainerStarted","Data":"b4eb3a1748892480ef22db18957f0d3c9d09701f282638b700b1f37d44c08e16"} Sep 16 00:56:28.466078 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:28.466008 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-228a5f03-2v9vh" event={"ID":"f1e1b23e-bbdc-4253-9625-90b8a3315ef9","Type":"ContainerStarted","Data":"ac536f34f784c34deca43b31b6086b76131f8870204eacef2659c51e5df58209"} Sep 16 00:56:28.482021 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:28.481959 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-228a5f03-2v9vh" podStartSLOduration=1.481942216 podStartE2EDuration="1.481942216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:56:28.479794316 +0000 UTC m=+1338.842017870" watchObservedRunningTime="2026-09-16 00:56:28.481942216 +0000 UTC m=+1338.844165748" Sep 16 00:56:28.996738 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:28.996697 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:29.474012 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:29.473977 2567 generic.go:356] "Generic (PLEG): container finished" podID="f1e1b23e-bbdc-4253-9625-90b8a3315ef9" containerID="b4eb3a1748892480ef22db18957f0d3c9d09701f282638b700b1f37d44c08e16" exitCode=0 Sep 16 00:56:29.474562 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:29.474069 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-228a5f03-2v9vh" event={"ID":"f1e1b23e-bbdc-4253-9625-90b8a3315ef9","Type":"ContainerDied","Data":"b4eb3a1748892480ef22db18957f0d3c9d09701f282638b700b1f37d44c08e16"} Sep 16 00:56:29.823639 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:29.823604 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:30.550096 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:30.550056 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:56:30.612857 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:30.612828 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-228a5f03-2v9vh" Sep 16 00:56:30.616656 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:30.616627 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e1b23e-bbdc-4253-9625-90b8a3315ef9-maas-api-service-ca\") pod \"f1e1b23e-bbdc-4253-9625-90b8a3315ef9\" (UID: \"f1e1b23e-bbdc-4253-9625-90b8a3315ef9\") " Sep 16 00:56:30.616971 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:30.616949 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1e1b23e-bbdc-4253-9625-90b8a3315ef9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f1e1b23e-bbdc-4253-9625-90b8a3315ef9" (UID: "f1e1b23e-bbdc-4253-9625-90b8a3315ef9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:56:30.717931 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:30.717870 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e1b23e-bbdc-4253-9625-90b8a3315ef9-maas-api-service-ca\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:56:31.487512 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:31.487434 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-228a5f03-2v9vh" Sep 16 00:56:31.487512 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:31.487460 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-228a5f03-2v9vh" event={"ID":"f1e1b23e-bbdc-4253-9625-90b8a3315ef9","Type":"ContainerDied","Data":"ac536f34f784c34deca43b31b6086b76131f8870204eacef2659c51e5df58209"} Sep 16 00:56:31.487512 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:31.487504 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac536f34f784c34deca43b31b6086b76131f8870204eacef2659c51e5df58209" Sep 16 00:56:32.670026 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:32.669985 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:37.803791 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:37.803740 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:37.870542 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:37.870512 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:37.929926 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:37.929886 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:37.999087 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:37.998994 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-228a5f03-2v9vh"] Sep 16 00:56:37.999881 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:37.999837 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:38.001462 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:38.001440 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-228a5f03-2v9vh"] Sep 16 00:56:38.062693 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:38.062593 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:38.122383 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:38.122332 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1e1b23e-bbdc-4253-9625-90b8a3315ef9" path="/var/lib/kubelet/pods/f1e1b23e-bbdc-4253-9625-90b8a3315ef9/volumes" Sep 16 00:56:38.390466 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:38.390351 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:38.447206 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:38.447175 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:39.158930 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:39.158897 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:43.744076 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:43.744021 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:56:44.064773 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:44.064736 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:44.163704 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:44.163669 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:44.332424 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:44.332309 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:44.699705 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:44.699610 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:44.799060 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:44.799007 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:44.934455 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:44.934417 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:45.401548 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:45.401509 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:45.502963 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:45.502914 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:50.906525 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:50.906486 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:50.982024 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:50.981983 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:51.071117 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:51.071079 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:51.166223 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:51.166121 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:51.265688 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:51.265630 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:53.052060 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:53.052011 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:53.989603 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:53.989562 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw"] Sep 16 00:56:54.002583 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:54.002532 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:56.551128 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:56.551081 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:56:57.578680 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:57.578642 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:57.679288 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:57.679256 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:58.515905 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:58.515873 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:58.824407 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:58.824357 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:56:59.110015 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:56:59.109925 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:01.626262 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:01.626216 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:02.127357 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:02.127313 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:02.221936 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:02.221906 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:02.355883 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:02.355846 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:03.456373 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:03.456330 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:03.585434 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:03.585388 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:03.685297 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:03.685256 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:04.078389 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:04.078355 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:04.178472 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:04.178435 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:09.010523 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.010448 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" podUID="c294a8d7-a9f7-448c-8c69-495698d7da1c" containerName="main" containerID="cri-o://4e9de3aa5df7e40a94a698bc53514d92d1eb791307c9d3be5263875d81483b31" gracePeriod=45 Sep 16 00:57:09.243713 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.243675 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:57:09.272244 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.272172 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:57:09.378799 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.378754 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-home\") pod \"c294a8d7-a9f7-448c-8c69-495698d7da1c\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " Sep 16 00:57:09.379016 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.378830 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-model-cache\") pod \"c294a8d7-a9f7-448c-8c69-495698d7da1c\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " Sep 16 00:57:09.379016 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.378910 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-tmp-dir\") pod \"c294a8d7-a9f7-448c-8c69-495698d7da1c\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " Sep 16 00:57:09.379016 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.378966 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-dshm\") pod \"c294a8d7-a9f7-448c-8c69-495698d7da1c\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " Sep 16 00:57:09.379016 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.378992 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c294a8d7-a9f7-448c-8c69-495698d7da1c-tls-certs\") pod \"c294a8d7-a9f7-448c-8c69-495698d7da1c\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " Sep 16 00:57:09.379267 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.379018 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nshl\" (UniqueName: \"kubernetes.io/projected/c294a8d7-a9f7-448c-8c69-495698d7da1c-kube-api-access-7nshl\") pod \"c294a8d7-a9f7-448c-8c69-495698d7da1c\" (UID: \"c294a8d7-a9f7-448c-8c69-495698d7da1c\") " Sep 16 00:57:09.379267 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.379101 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-home" (OuterVolumeSpecName: "home") pod "c294a8d7-a9f7-448c-8c69-495698d7da1c" (UID: "c294a8d7-a9f7-448c-8c69-495698d7da1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:57:09.379267 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.379136 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-model-cache" (OuterVolumeSpecName: "model-cache") pod "c294a8d7-a9f7-448c-8c69-495698d7da1c" (UID: "c294a8d7-a9f7-448c-8c69-495698d7da1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:57:09.379267 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.379196 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c294a8d7-a9f7-448c-8c69-495698d7da1c" (UID: "c294a8d7-a9f7-448c-8c69-495698d7da1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:57:09.379592 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.379558 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-tmp-dir\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:57:09.379592 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.379585 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-home\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:57:09.379592 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.379597 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-model-cache\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:57:09.381573 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.381539 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-dshm" (OuterVolumeSpecName: "dshm") pod "c294a8d7-a9f7-448c-8c69-495698d7da1c" (UID: "c294a8d7-a9f7-448c-8c69-495698d7da1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:57:09.381681 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.381590 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c294a8d7-a9f7-448c-8c69-495698d7da1c-kube-api-access-7nshl" (OuterVolumeSpecName: "kube-api-access-7nshl") pod "c294a8d7-a9f7-448c-8c69-495698d7da1c" (UID: "c294a8d7-a9f7-448c-8c69-495698d7da1c"). InnerVolumeSpecName "kube-api-access-7nshl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:57:09.381681 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.381651 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c294a8d7-a9f7-448c-8c69-495698d7da1c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c294a8d7-a9f7-448c-8c69-495698d7da1c" (UID: "c294a8d7-a9f7-448c-8c69-495698d7da1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 00:57:09.480798 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.480757 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c294a8d7-a9f7-448c-8c69-495698d7da1c-dshm\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:57:09.480798 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.480794 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c294a8d7-a9f7-448c-8c69-495698d7da1c-tls-certs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:57:09.480798 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.480805 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nshl\" (UniqueName: \"kubernetes.io/projected/c294a8d7-a9f7-448c-8c69-495698d7da1c-kube-api-access-7nshl\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:57:09.722896 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.722858 2567 generic.go:356] "Generic (PLEG): container finished" podID="c294a8d7-a9f7-448c-8c69-495698d7da1c" containerID="4e9de3aa5df7e40a94a698bc53514d92d1eb791307c9d3be5263875d81483b31" exitCode=0 Sep 16 00:57:09.723142 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.722906 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" Sep 16 00:57:09.723142 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.722927 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" event={"ID":"c294a8d7-a9f7-448c-8c69-495698d7da1c","Type":"ContainerDied","Data":"4e9de3aa5df7e40a94a698bc53514d92d1eb791307c9d3be5263875d81483b31"} Sep 16 00:57:09.723142 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.722968 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw" event={"ID":"c294a8d7-a9f7-448c-8c69-495698d7da1c","Type":"ContainerDied","Data":"8e5ca492ed1033874a2469fc4c4eb72b27094e851c8c10eb32e527b212939c27"} Sep 16 00:57:09.723142 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.722991 2567 scope.go:117] "RemoveContainer" containerID="4e9de3aa5df7e40a94a698bc53514d92d1eb791307c9d3be5263875d81483b31" Sep 16 00:57:09.733451 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.733426 2567 scope.go:117] "RemoveContainer" containerID="4e9de3aa5df7e40a94a698bc53514d92d1eb791307c9d3be5263875d81483b31" Sep 16 00:57:09.733776 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:57:09.733754 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e9de3aa5df7e40a94a698bc53514d92d1eb791307c9d3be5263875d81483b31\": container with ID starting with 4e9de3aa5df7e40a94a698bc53514d92d1eb791307c9d3be5263875d81483b31 not found: ID does not exist" containerID="4e9de3aa5df7e40a94a698bc53514d92d1eb791307c9d3be5263875d81483b31" Sep 16 00:57:09.733832 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.733789 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e9de3aa5df7e40a94a698bc53514d92d1eb791307c9d3be5263875d81483b31"} err="failed to get container status \"4e9de3aa5df7e40a94a698bc53514d92d1eb791307c9d3be5263875d81483b31\": rpc error: code = NotFound desc = could not find container \"4e9de3aa5df7e40a94a698bc53514d92d1eb791307c9d3be5263875d81483b31\": container with ID starting with 4e9de3aa5df7e40a94a698bc53514d92d1eb791307c9d3be5263875d81483b31 not found: ID does not exist" Sep 16 00:57:09.746810 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.746773 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw"] Sep 16 00:57:09.749199 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:09.749170 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2fa86d/auth-test-model-2fa86d-kserve-9d8897cbc-fz6lw"] Sep 16 00:57:10.119259 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:10.119223 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c294a8d7-a9f7-448c-8c69-495698d7da1c" path="/var/lib/kubelet/pods/c294a8d7-a9f7-448c-8c69-495698d7da1c/volumes" Sep 16 00:57:10.253708 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:10.253673 2567 scope.go:117] "RemoveContainer" containerID="84d3f8ea36b69ec9a942012e61e4d0fae5fd1dff57029d434480c3da018728da" Sep 16 00:57:12.540654 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:12.540614 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:15.318616 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.318570 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b"] Sep 16 00:57:15.319191 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.318982 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c294a8d7-a9f7-448c-8c69-495698d7da1c" containerName="main" Sep 16 00:57:15.319191 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.318993 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c294a8d7-a9f7-448c-8c69-495698d7da1c" containerName="main" Sep 16 00:57:15.319191 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.319051 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1e1b23e-bbdc-4253-9625-90b8a3315ef9" containerName="revoke-keys" Sep 16 00:57:15.319191 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.319058 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1e1b23e-bbdc-4253-9625-90b8a3315ef9" containerName="revoke-keys" Sep 16 00:57:15.319191 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.319119 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eff9577-aa6f-4e10-8538-ba0d09f132eb" containerName="istio-proxy" Sep 16 00:57:15.319191 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.319130 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c294a8d7-a9f7-448c-8c69-495698d7da1c" containerName="main" Sep 16 00:57:15.319191 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.319135 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1e1b23e-bbdc-4253-9625-90b8a3315ef9" containerName="revoke-keys" Sep 16 00:57:15.324087 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.324060 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:15.328280 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.328243 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-108960\"/\"kube-root-ca.crt\"" Sep 16 00:57:15.328463 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.328243 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-108960\"/\"openshift-service-ca.crt\"" Sep 16 00:57:15.328463 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.328243 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-108960\"/\"default-dockercfg-9wl2p\"" Sep 16 00:57:15.328463 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.328377 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-108960\"/\"sub-test-model-108960-kserve-self-signed-certs\"" Sep 16 00:57:15.332197 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.332167 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b"] Sep 16 00:57:15.337695 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.337667 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:15.437993 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.437945 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abfb4e62-c537-4fb6-bc86-2a2ec0295113-tls-certs\") pod \"sub-test-model-108960-kserve-5985dcc8dc-q8h7b\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:15.437993 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.437993 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-dshm\") pod \"sub-test-model-108960-kserve-5985dcc8dc-q8h7b\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:15.438280 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.438021 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-tmp-dir\") pod \"sub-test-model-108960-kserve-5985dcc8dc-q8h7b\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:15.438280 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.438108 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggkm6\" (UniqueName: \"kubernetes.io/projected/abfb4e62-c537-4fb6-bc86-2a2ec0295113-kube-api-access-ggkm6\") pod \"sub-test-model-108960-kserve-5985dcc8dc-q8h7b\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:15.438280 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.438147 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-model-cache\") pod \"sub-test-model-108960-kserve-5985dcc8dc-q8h7b\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:15.438280 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.438177 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-home\") pod \"sub-test-model-108960-kserve-5985dcc8dc-q8h7b\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:15.538974 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.538926 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-dshm\") pod \"sub-test-model-108960-kserve-5985dcc8dc-q8h7b\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:15.539247 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.538985 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-tmp-dir\") pod \"sub-test-model-108960-kserve-5985dcc8dc-q8h7b\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:15.539247 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.539028 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggkm6\" (UniqueName: \"kubernetes.io/projected/abfb4e62-c537-4fb6-bc86-2a2ec0295113-kube-api-access-ggkm6\") pod \"sub-test-model-108960-kserve-5985dcc8dc-q8h7b\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:15.539247 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.539118 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-model-cache\") pod \"sub-test-model-108960-kserve-5985dcc8dc-q8h7b\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:15.539247 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.539162 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-home\") pod \"sub-test-model-108960-kserve-5985dcc8dc-q8h7b\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:15.539247 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.539223 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abfb4e62-c537-4fb6-bc86-2a2ec0295113-tls-certs\") pod \"sub-test-model-108960-kserve-5985dcc8dc-q8h7b\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:15.539545 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.539471 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-tmp-dir\") pod \"sub-test-model-108960-kserve-5985dcc8dc-q8h7b\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:15.539545 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.539513 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-model-cache\") pod \"sub-test-model-108960-kserve-5985dcc8dc-q8h7b\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:15.539654 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.539550 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-home\") pod \"sub-test-model-108960-kserve-5985dcc8dc-q8h7b\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:15.541483 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.541462 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-dshm\") pod \"sub-test-model-108960-kserve-5985dcc8dc-q8h7b\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:15.541795 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.541776 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abfb4e62-c537-4fb6-bc86-2a2ec0295113-tls-certs\") pod \"sub-test-model-108960-kserve-5985dcc8dc-q8h7b\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:15.548253 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.548224 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggkm6\" (UniqueName: \"kubernetes.io/projected/abfb4e62-c537-4fb6-bc86-2a2ec0295113-kube-api-access-ggkm6\") pod \"sub-test-model-108960-kserve-5985dcc8dc-q8h7b\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:15.636087 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.635941 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:15.912257 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:15.912224 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b"] Sep 16 00:57:15.914260 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:57:15.914233 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabfb4e62_c537_4fb6_bc86_2a2ec0295113.slice/crio-cfa1803ce1f59d8519f5c2080ad3644a13cc50330220033845df6864ed4c3478 WatchSource:0}: Error finding container cfa1803ce1f59d8519f5c2080ad3644a13cc50330220033845df6864ed4c3478: Status 404 returned error can't find the container with id cfa1803ce1f59d8519f5c2080ad3644a13cc50330220033845df6864ed4c3478 Sep 16 00:57:16.768265 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:16.768219 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" event={"ID":"abfb4e62-c537-4fb6-bc86-2a2ec0295113","Type":"ContainerStarted","Data":"714a6a19569d1feb1676c0f3ca5120ccd932504b49f5a4fe164c8ec44d74b2c1"} Sep 16 00:57:16.768265 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:16.768270 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" event={"ID":"abfb4e62-c537-4fb6-bc86-2a2ec0295113","Type":"ContainerStarted","Data":"cfa1803ce1f59d8519f5c2080ad3644a13cc50330220033845df6864ed4c3478"} Sep 16 00:57:16.792538 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:16.787338 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" podStartSLOduration=1.787314654 podStartE2EDuration="1.787314654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:57:16.783651782 +0000 UTC m=+1387.145875334" watchObservedRunningTime="2026-09-16 00:57:16.787314654 +0000 UTC m=+1387.149538187" Sep 16 00:57:18.149829 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:18.149782 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:57:24.462852 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:24.462812 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:25.636195 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:25.636150 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:25.636704 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:25.636213 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:25.649350 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:25.649319 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:25.832835 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:25.832803 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:57:26.039613 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:26.039523 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:26.929199 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:26.929153 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:27.029979 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.029945 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:27.377359 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.377270 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:27.660360 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.660267 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l"] Sep 16 00:57:27.663941 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.663915 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.666970 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.666946 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-409cde-openshift-default-dockercfg-dbd2w\"" Sep 16 00:57:27.678902 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.678866 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l"] Sep 16 00:57:27.695585 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.695549 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:27.755910 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.755868 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-envoy\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.756186 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.755963 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-podinfo\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.756186 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.756015 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-workload-certs\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.756186 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.756057 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8bvv\" (UniqueName: \"kubernetes.io/projected/177f8426-32c4-43a9-b689-7984a4b2cd0f-kube-api-access-n8bvv\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.756186 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.756082 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-token\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.756186 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.756118 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-workload-socket\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.756186 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.756143 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-credential-socket\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.756186 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.756174 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177f8426-32c4-43a9-b689-7984a4b2cd0f-istiod-ca-cert\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.756569 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.756213 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-data\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.790531 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.790489 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:27.857685 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.857630 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-workload-certs\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.857685 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.857686 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8bvv\" (UniqueName: \"kubernetes.io/projected/177f8426-32c4-43a9-b689-7984a4b2cd0f-kube-api-access-n8bvv\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.857963 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.857810 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-token\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.857963 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.857859 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-workload-socket\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.858127 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.857974 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-credential-socket\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.858127 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.858026 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177f8426-32c4-43a9-b689-7984a4b2cd0f-istiod-ca-cert\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.858127 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.858117 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-workload-certs\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.858289 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.858143 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-data\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.858289 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.858219 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-workload-socket\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.858289 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.858250 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-envoy\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.858449 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.858306 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-podinfo\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.858449 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.858377 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-credential-socket\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.858449 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.858407 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-data\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.858771 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.858747 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177f8426-32c4-43a9-b689-7984a4b2cd0f-istiod-ca-cert\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.860589 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.860568 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-envoy\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.860821 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.860803 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-podinfo\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.866458 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.866431 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-token\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.866603 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.866566 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8bvv\" (UniqueName: \"kubernetes.io/projected/177f8426-32c4-43a9-b689-7984a4b2cd0f-kube-api-access-n8bvv\") pod \"e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:27.977717 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:27.977614 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:28.038502 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:28.038471 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:28.271397 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:28.271180 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l"] Sep 16 00:57:28.274330 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:57:28.274294 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177f8426_32c4_43a9_b689_7984a4b2cd0f.slice/crio-b815ea5da96a78dbc4188f742bce81e1cba3cd4ef8f8b30c202ecc10eb433111 WatchSource:0}: Error finding container b815ea5da96a78dbc4188f742bce81e1cba3cd4ef8f8b30c202ecc10eb433111: Status 404 returned error can't find the container with id b815ea5da96a78dbc4188f742bce81e1cba3cd4ef8f8b30c202ecc10eb433111 Sep 16 00:57:28.276779 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:28.276733 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:57:28.276889 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:28.276816 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:57:28.276889 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:28.276854 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:57:28.841628 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:28.841591 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" event={"ID":"177f8426-32c4-43a9-b689-7984a4b2cd0f","Type":"ContainerStarted","Data":"735d94fd1533eba001fec7b939d8de647c4cc95b278cb6b1bd77e39467ca4255"} Sep 16 00:57:28.841821 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:28.841647 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" event={"ID":"177f8426-32c4-43a9-b689-7984a4b2cd0f","Type":"ContainerStarted","Data":"b815ea5da96a78dbc4188f742bce81e1cba3cd4ef8f8b30c202ecc10eb433111"} Sep 16 00:57:28.865478 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:28.865409 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" podStartSLOduration=1.865387106 podStartE2EDuration="1.865387106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:57:28.864417004 +0000 UTC m=+1399.226640556" watchObservedRunningTime="2026-09-16 00:57:28.865387106 +0000 UTC m=+1399.227610637" Sep 16 00:57:28.978735 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:28.978647 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:29.983858 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:29.983824 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:30.307090 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:30.307055 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:30.800667 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:30.800626 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:30.854328 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:30.854290 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:30.855593 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:30.855569 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:30.882814 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:30.882778 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:31.191925 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:31.191820 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:32.147770 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:32.147735 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:32.823435 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:32.823402 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:32.878947 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:32.878914 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:33.274384 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:33.274267 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:33.373742 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:33.373713 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:33.564329 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:33.564289 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:33.935333 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:33.935241 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:39.342586 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:39.342532 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:57:43.561229 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:43.561181 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:50.983164 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:50.983129 2567 generic.go:356] "Generic (PLEG): container finished" podID="177f8426-32c4-43a9-b689-7984a4b2cd0f" containerID="735d94fd1533eba001fec7b939d8de647c4cc95b278cb6b1bd77e39467ca4255" exitCode=0 Sep 16 00:57:50.983609 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:50.983214 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" event={"ID":"177f8426-32c4-43a9-b689-7984a4b2cd0f","Type":"ContainerDied","Data":"735d94fd1533eba001fec7b939d8de647c4cc95b278cb6b1bd77e39467ca4255"} Sep 16 00:57:50.983664 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:50.983649 2567 scope.go:117] "RemoveContainer" containerID="735d94fd1533eba001fec7b939d8de647c4cc95b278cb6b1bd77e39467ca4255" Sep 16 00:57:50.984819 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:50.984786 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:57:50.984925 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:50.984861 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:57:50.984925 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:50.984893 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:57:51.991221 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:51.991177 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" event={"ID":"177f8426-32c4-43a9-b689-7984a4b2cd0f","Type":"ContainerStarted","Data":"9d50fbd075c513bcf16d120047ea5b37ead86c790eeabc53a8de2b7e9f034a6d"} Sep 16 00:57:52.978528 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:52.978483 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:52.978804 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:52.978780 2567 patch_prober.go:28] interesting pod/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 16 00:57:52.978864 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:52.978834 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" podUID="177f8426-32c4-43a9-b689-7984a4b2cd0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 16 00:57:53.647982 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:53.647923 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:53.978639 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:53.978548 2567 patch_prober.go:28] interesting pod/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 16 00:57:53.978639 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:53.978610 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" podUID="177f8426-32c4-43a9-b689-7984a4b2cd0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 16 00:57:54.978085 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:54.978024 2567 patch_prober.go:28] interesting pod/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 16 00:57:54.978527 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:54.978110 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" podUID="177f8426-32c4-43a9-b689-7984a4b2cd0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 16 00:57:54.981374 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:54.981346 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:55.793013 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:55.792981 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:55.922241 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:55.922209 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:55.978318 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:55.978277 2567 patch_prober.go:28] interesting pod/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 16 00:57:55.978725 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:55.978353 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" podUID="177f8426-32c4-43a9-b689-7984a4b2cd0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 16 00:57:56.610963 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:56.610929 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:56.979013 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:56.978920 2567 patch_prober.go:28] interesting pod/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 16 00:57:56.979013 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:56.978982 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" podUID="177f8426-32c4-43a9-b689-7984a4b2cd0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 16 00:57:57.221892 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:57.221859 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:57:57.983539 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:57.983496 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:57.983939 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:57.983808 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:57.984719 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:57.984698 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 00:57:58.235180 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:57:58.235084 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:58:00.499700 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:00.499665 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9358-24d20f"] Sep 16 00:58:00.502300 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:00.502279 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9358-24d20f" Sep 16 00:58:00.505011 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:00.504984 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qndhv\"" Sep 16 00:58:00.510850 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:00.510633 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9358-24d20f"] Sep 16 00:58:00.600205 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:00.600171 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7dng\" (UniqueName: \"kubernetes.io/projected/dfcf2915-daa3-4401-8b01-bbc3aa459d39-kube-api-access-z7dng\") pod \"test-curl-9358-24d20f\" (UID: \"dfcf2915-daa3-4401-8b01-bbc3aa459d39\") " pod="opendatahub/test-curl-9358-24d20f" Sep 16 00:58:00.701160 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:00.701119 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7dng\" (UniqueName: \"kubernetes.io/projected/dfcf2915-daa3-4401-8b01-bbc3aa459d39-kube-api-access-z7dng\") pod \"test-curl-9358-24d20f\" (UID: \"dfcf2915-daa3-4401-8b01-bbc3aa459d39\") " pod="opendatahub/test-curl-9358-24d20f" Sep 16 00:58:00.710203 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:00.710174 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7dng\" (UniqueName: \"kubernetes.io/projected/dfcf2915-daa3-4401-8b01-bbc3aa459d39-kube-api-access-z7dng\") pod \"test-curl-9358-24d20f\" (UID: \"dfcf2915-daa3-4401-8b01-bbc3aa459d39\") " pod="opendatahub/test-curl-9358-24d20f" Sep 16 00:58:00.814228 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:00.814187 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9358-24d20f" Sep 16 00:58:01.088406 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:01.088372 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9358-24d20f"] Sep 16 00:58:01.091523 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:58:01.091492 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfcf2915_daa3_4401_8b01_bbc3aa459d39.slice/crio-5d57a19899dd8ff41e531113c3aab76036665cc365361da260d1a470ddb75458 WatchSource:0}: Error finding container 5d57a19899dd8ff41e531113c3aab76036665cc365361da260d1a470ddb75458: Status 404 returned error can't find the container with id 5d57a19899dd8ff41e531113c3aab76036665cc365361da260d1a470ddb75458 Sep 16 00:58:01.093891 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:01.093873 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 00:58:02.052240 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:02.052202 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9358-24d20f" event={"ID":"dfcf2915-daa3-4401-8b01-bbc3aa459d39","Type":"ContainerStarted","Data":"a437e757ca08d7e57a814fc5026b625c779c6821bb94b6b5e6c307f3f9ea9335"} Sep 16 00:58:02.052240 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:02.052249 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9358-24d20f" event={"ID":"dfcf2915-daa3-4401-8b01-bbc3aa459d39","Type":"ContainerStarted","Data":"5d57a19899dd8ff41e531113c3aab76036665cc365361da260d1a470ddb75458"} Sep 16 00:58:02.067360 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:02.067289 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9358-24d20f" podStartSLOduration=1.493551992 podStartE2EDuration="2.067271425s" totalImagesPullingTime="573.719433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:58:02.066095776 +0000 UTC m=+1432.428319308" watchObservedRunningTime="2026-09-16 00:58:02.067271425 +0000 UTC m=+1432.429494956" Sep 16 00:58:02.704536 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:02.704494 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9358-24d20f"] Sep 16 00:58:02.709161 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:02.709116 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9358-24d20f"] Sep 16 00:58:03.058016 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:58:03.057984 2567 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9358-24d20f" container="test-curl-9358-24d20f" Sep 16 00:58:04.063653 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:04.063609 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9358-24d20f" podUID="dfcf2915-daa3-4401-8b01-bbc3aa459d39" containerName="test-curl-9358-24d20f" containerID="cri-o://a437e757ca08d7e57a814fc5026b625c779c6821bb94b6b5e6c307f3f9ea9335" gracePeriod=2 Sep 16 00:58:04.066126 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:58:04.066091 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9358-24d20f\" is forbidden: User \"system:node:ip-10-0-131-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-194.ec2.internal' and this object" podUID="dfcf2915-daa3-4401-8b01-bbc3aa459d39" pod="opendatahub/test-curl-9358-24d20f" Sep 16 00:58:06.255122 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:06.255093 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9358-24d20f" Sep 16 00:58:06.358484 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:06.358382 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7dng\" (UniqueName: \"kubernetes.io/projected/dfcf2915-daa3-4401-8b01-bbc3aa459d39-kube-api-access-z7dng\") pod \"dfcf2915-daa3-4401-8b01-bbc3aa459d39\" (UID: \"dfcf2915-daa3-4401-8b01-bbc3aa459d39\") " Sep 16 00:58:06.360970 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:06.360945 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfcf2915-daa3-4401-8b01-bbc3aa459d39-kube-api-access-z7dng" (OuterVolumeSpecName: "kube-api-access-z7dng") pod "dfcf2915-daa3-4401-8b01-bbc3aa459d39" (UID: "dfcf2915-daa3-4401-8b01-bbc3aa459d39"). InnerVolumeSpecName "kube-api-access-z7dng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:58:06.459815 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:06.459777 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7dng\" (UniqueName: \"kubernetes.io/projected/dfcf2915-daa3-4401-8b01-bbc3aa459d39-kube-api-access-z7dng\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:58:06.732771 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:06.732679 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:07.016301 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.016163 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn"] Sep 16 00:58:07.016817 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.016792 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfcf2915-daa3-4401-8b01-bbc3aa459d39" containerName="test-curl-9358-24d20f" Sep 16 00:58:07.016913 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.016821 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfcf2915-daa3-4401-8b01-bbc3aa459d39" containerName="test-curl-9358-24d20f" Sep 16 00:58:07.016957 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.016932 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfcf2915-daa3-4401-8b01-bbc3aa459d39" containerName="test-curl-9358-24d20f" Sep 16 00:58:07.019549 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.019518 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.022337 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.022311 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-37b46587-openshift-default-dockercfg-chs4p\"" Sep 16 00:58:07.031753 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.031725 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn"] Sep 16 00:58:07.044050 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.044003 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:07.066115 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.066064 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-credential-socket\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.066115 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.066117 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-data\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.066346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.066194 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-workload-socket\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.066346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.066241 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-workload-certs\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.066346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.066262 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-token\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.066346 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.066322 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-podinfo\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.066486 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.066392 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpxrt\" (UniqueName: \"kubernetes.io/projected/6b00f081-7f0d-471b-bdd4-85bfe427477f-kube-api-access-wpxrt\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.066486 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.066429 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-envoy\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.066486 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.066451 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b00f081-7f0d-471b-bdd4-85bfe427477f-istiod-ca-cert\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.084758 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.084714 2567 generic.go:356] "Generic (PLEG): container finished" podID="dfcf2915-daa3-4401-8b01-bbc3aa459d39" containerID="a437e757ca08d7e57a814fc5026b625c779c6821bb94b6b5e6c307f3f9ea9335" exitCode=137 Sep 16 00:58:07.084934 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.084769 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9358-24d20f" Sep 16 00:58:07.084934 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.084827 2567 scope.go:117] "RemoveContainer" containerID="a437e757ca08d7e57a814fc5026b625c779c6821bb94b6b5e6c307f3f9ea9335" Sep 16 00:58:07.096361 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.096342 2567 scope.go:117] "RemoveContainer" containerID="a437e757ca08d7e57a814fc5026b625c779c6821bb94b6b5e6c307f3f9ea9335" Sep 16 00:58:07.096751 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:58:07.096725 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a437e757ca08d7e57a814fc5026b625c779c6821bb94b6b5e6c307f3f9ea9335\": container with ID starting with a437e757ca08d7e57a814fc5026b625c779c6821bb94b6b5e6c307f3f9ea9335 not found: ID does not exist" containerID="a437e757ca08d7e57a814fc5026b625c779c6821bb94b6b5e6c307f3f9ea9335" Sep 16 00:58:07.096808 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.096763 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a437e757ca08d7e57a814fc5026b625c779c6821bb94b6b5e6c307f3f9ea9335"} err="failed to get container status \"a437e757ca08d7e57a814fc5026b625c779c6821bb94b6b5e6c307f3f9ea9335\": rpc error: code = NotFound desc = could not find container \"a437e757ca08d7e57a814fc5026b625c779c6821bb94b6b5e6c307f3f9ea9335\": container with ID starting with a437e757ca08d7e57a814fc5026b625c779c6821bb94b6b5e6c307f3f9ea9335 not found: ID does not exist" Sep 16 00:58:07.097325 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:58:07.097304 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9358-24d20f\" is forbidden: User \"system:node:ip-10-0-131-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-194.ec2.internal' and this object" podUID="dfcf2915-daa3-4401-8b01-bbc3aa459d39" pod="opendatahub/test-curl-9358-24d20f" Sep 16 00:58:07.167013 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.166962 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-workload-socket\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.167013 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.167014 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-workload-certs\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.167295 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.167047 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-token\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.167295 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.167117 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-podinfo\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.167295 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.167170 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpxrt\" (UniqueName: \"kubernetes.io/projected/6b00f081-7f0d-471b-bdd4-85bfe427477f-kube-api-access-wpxrt\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.167295 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.167200 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-envoy\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.167295 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.167220 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b00f081-7f0d-471b-bdd4-85bfe427477f-istiod-ca-cert\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.167667 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.167344 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-credential-socket\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.167667 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.167385 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-data\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.167667 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.167515 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-workload-socket\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.167667 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.167554 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-workload-certs\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.167897 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.167722 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-credential-socket\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.167967 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.167946 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-data\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.168236 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.168215 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b00f081-7f0d-471b-bdd4-85bfe427477f-istiod-ca-cert\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.170032 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.169998 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-podinfo\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.170163 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.170103 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-envoy\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.175332 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.175297 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-token\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.175662 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.175638 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpxrt\" (UniqueName: \"kubernetes.io/projected/6b00f081-7f0d-471b-bdd4-85bfe427477f-kube-api-access-wpxrt\") pod \"e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.316476 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.316438 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:07.331219 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.331179 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:07.618005 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.617976 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn"] Sep 16 00:58:07.620162 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:58:07.620124 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b00f081_7f0d_471b_bdd4_85bfe427477f.slice/crio-518eb0c872c574e238f88ee6235843e4bb11278af78f78e6f4e9aab3903b2a97 WatchSource:0}: Error finding container 518eb0c872c574e238f88ee6235843e4bb11278af78f78e6f4e9aab3903b2a97: Status 404 returned error can't find the container with id 518eb0c872c574e238f88ee6235843e4bb11278af78f78e6f4e9aab3903b2a97 Sep 16 00:58:07.622949 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.622912 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:58:07.623099 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.622994 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:58:07.623099 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:07.623027 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:58:08.092630 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:08.092586 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" event={"ID":"6b00f081-7f0d-471b-bdd4-85bfe427477f","Type":"ContainerStarted","Data":"e12ee3610eb098c5b4beeee813475f8bcbf5112ade9ebb1c2227459c20ba8d71"} Sep 16 00:58:08.092814 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:08.092662 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" event={"ID":"6b00f081-7f0d-471b-bdd4-85bfe427477f","Type":"ContainerStarted","Data":"518eb0c872c574e238f88ee6235843e4bb11278af78f78e6f4e9aab3903b2a97"} Sep 16 00:58:08.115204 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:58:08.115166 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9358-24d20f\" is forbidden: User \"system:node:ip-10-0-131-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-194.ec2.internal' and this object" podUID="dfcf2915-daa3-4401-8b01-bbc3aa459d39" pod="opendatahub/test-curl-9358-24d20f" Sep 16 00:58:08.116294 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:08.116247 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" podStartSLOduration=2.116229073 podStartE2EDuration="2.116229073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:58:08.112992433 +0000 UTC m=+1438.475216039" watchObservedRunningTime="2026-09-16 00:58:08.116229073 +0000 UTC m=+1438.478452710" Sep 16 00:58:08.120270 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:08.120238 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfcf2915-daa3-4401-8b01-bbc3aa459d39" path="/var/lib/kubelet/pods/dfcf2915-daa3-4401-8b01-bbc3aa459d39/volumes" Sep 16 00:58:08.331756 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:08.331718 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:08.337687 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:08.337655 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:09.100090 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:09.100031 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:09.101281 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:09.101262 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:09.443956 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:09.443866 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:10.022355 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:10.022316 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:10.112303 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:10.112242 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:10.281753 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:10.281645 2567 scope.go:117] "RemoveContainer" containerID="96ef4f71214b32348289235a0175563280e4b25063c868fdff4a046cb2e554a4" Sep 16 00:58:10.363231 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:10.363197 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:11.139703 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:11.139670 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:12.033281 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:12.033248 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:12.694242 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:12.694199 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:12.795477 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:12.795446 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:13.097385 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.097350 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:13.195786 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.195747 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:13.267631 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.267588 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:13.268977 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.268947 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s"] Sep 16 00:58:13.272333 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.272309 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s" Sep 16 00:58:13.275169 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.275147 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-37b46587-dockercfg-79cmk\"" Sep 16 00:58:13.275169 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.275162 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-37b46587\"" Sep 16 00:58:13.285519 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.285490 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s"] Sep 16 00:58:13.305686 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.305648 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78"] Sep 16 00:58:13.308566 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.308547 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78" Sep 16 00:58:13.320600 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.320570 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78"] Sep 16 00:58:13.335398 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.335360 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07adb093-0a0b-4c76-b510-a5b54e8a3f19-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s\" (UID: \"07adb093-0a0b-4c76-b510-a5b54e8a3f19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s" Sep 16 00:58:13.335574 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.335443 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf74p\" (UniqueName: \"kubernetes.io/projected/07adb093-0a0b-4c76-b510-a5b54e8a3f19-kube-api-access-gf74p\") pod \"payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s\" (UID: \"07adb093-0a0b-4c76-b510-a5b54e8a3f19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s" Sep 16 00:58:13.436630 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.436534 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07adb093-0a0b-4c76-b510-a5b54e8a3f19-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s\" (UID: \"07adb093-0a0b-4c76-b510-a5b54e8a3f19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s" Sep 16 00:58:13.436630 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.436591 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7szdv\" (UniqueName: \"kubernetes.io/projected/cb4bf108-a2c2-40e0-a6da-d3ffa2368982-kube-api-access-7szdv\") pod \"payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78\" (UID: \"cb4bf108-a2c2-40e0-a6da-d3ffa2368982\") " pod="openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78" Sep 16 00:58:13.436833 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.436642 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf74p\" (UniqueName: \"kubernetes.io/projected/07adb093-0a0b-4c76-b510-a5b54e8a3f19-kube-api-access-gf74p\") pod \"payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s\" (UID: \"07adb093-0a0b-4c76-b510-a5b54e8a3f19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s" Sep 16 00:58:13.436833 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.436678 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4bf108-a2c2-40e0-a6da-d3ffa2368982-plugins-config-volume\") pod \"payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78\" (UID: \"cb4bf108-a2c2-40e0-a6da-d3ffa2368982\") " pod="openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78" Sep 16 00:58:13.437307 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.437287 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07adb093-0a0b-4c76-b510-a5b54e8a3f19-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s\" (UID: \"07adb093-0a0b-4c76-b510-a5b54e8a3f19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s" Sep 16 00:58:13.445796 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.445759 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf74p\" (UniqueName: \"kubernetes.io/projected/07adb093-0a0b-4c76-b510-a5b54e8a3f19-kube-api-access-gf74p\") pod \"payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s\" (UID: \"07adb093-0a0b-4c76-b510-a5b54e8a3f19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s" Sep 16 00:58:13.538321 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.538279 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7szdv\" (UniqueName: \"kubernetes.io/projected/cb4bf108-a2c2-40e0-a6da-d3ffa2368982-kube-api-access-7szdv\") pod \"payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78\" (UID: \"cb4bf108-a2c2-40e0-a6da-d3ffa2368982\") " pod="openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78" Sep 16 00:58:13.538533 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.538385 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4bf108-a2c2-40e0-a6da-d3ffa2368982-plugins-config-volume\") pod \"payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78\" (UID: \"cb4bf108-a2c2-40e0-a6da-d3ffa2368982\") " pod="openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78" Sep 16 00:58:13.539162 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.539140 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4bf108-a2c2-40e0-a6da-d3ffa2368982-plugins-config-volume\") pod \"payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78\" (UID: \"cb4bf108-a2c2-40e0-a6da-d3ffa2368982\") " pod="openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78" Sep 16 00:58:13.547225 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.547195 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7szdv\" (UniqueName: \"kubernetes.io/projected/cb4bf108-a2c2-40e0-a6da-d3ffa2368982-kube-api-access-7szdv\") pod \"payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78\" (UID: \"cb4bf108-a2c2-40e0-a6da-d3ffa2368982\") " pod="openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78" Sep 16 00:58:13.584452 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.584414 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s" Sep 16 00:58:13.620801 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.620762 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78" Sep 16 00:58:13.866417 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.866385 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s"] Sep 16 00:58:13.869012 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:58:13.868982 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07adb093_0a0b_4c76_b510_a5b54e8a3f19.slice/crio-e8c4b840a702787e98f31b62e5d6e9bc8f6df295439999f3b73e22f182a062f4 WatchSource:0}: Error finding container e8c4b840a702787e98f31b62e5d6e9bc8f6df295439999f3b73e22f182a062f4: Status 404 returned error can't find the container with id e8c4b840a702787e98f31b62e5d6e9bc8f6df295439999f3b73e22f182a062f4 Sep 16 00:58:13.912130 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:13.912105 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78"] Sep 16 00:58:13.914336 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:58:13.914306 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4bf108_a2c2_40e0_a6da_d3ffa2368982.slice/crio-c66b711d1d97c58b3a5aa4dfdefba2c3c41ca27529d8112596c6cd65edd1ecd7 WatchSource:0}: Error finding container c66b711d1d97c58b3a5aa4dfdefba2c3c41ca27529d8112596c6cd65edd1ecd7: Status 404 returned error can't find the container with id c66b711d1d97c58b3a5aa4dfdefba2c3c41ca27529d8112596c6cd65edd1ecd7 Sep 16 00:58:14.138795 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:14.138740 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s" event={"ID":"07adb093-0a0b-4c76-b510-a5b54e8a3f19","Type":"ContainerStarted","Data":"58179725b2b4c56163d18f3cd0c70dc13f3858a44b52f54c3524d7dd34135bec"} Sep 16 00:58:14.138974 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:14.138817 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s" event={"ID":"07adb093-0a0b-4c76-b510-a5b54e8a3f19","Type":"ContainerStarted","Data":"e8c4b840a702787e98f31b62e5d6e9bc8f6df295439999f3b73e22f182a062f4"} Sep 16 00:58:14.138974 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:14.138849 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s" Sep 16 00:58:14.140569 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:14.140538 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78" event={"ID":"cb4bf108-a2c2-40e0-a6da-d3ffa2368982","Type":"ContainerStarted","Data":"4c2e60d94043c59e5c188957a6484ec62d900574740a99965bc401c741083780"} Sep 16 00:58:14.140766 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:14.140590 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78" event={"ID":"cb4bf108-a2c2-40e0-a6da-d3ffa2368982","Type":"ContainerStarted","Data":"c66b711d1d97c58b3a5aa4dfdefba2c3c41ca27529d8112596c6cd65edd1ecd7"} Sep 16 00:58:14.140766 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:14.140617 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78" Sep 16 00:58:14.159455 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:14.159377 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s" podStartSLOduration=1.15935604 podStartE2EDuration="1.15935604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:58:14.156762823 +0000 UTC m=+1444.518986384" watchObservedRunningTime="2026-09-16 00:58:14.15935604 +0000 UTC m=+1444.521579572" Sep 16 00:58:14.175234 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:14.175179 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78" podStartSLOduration=1.175158193 podStartE2EDuration="1.175158193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:58:14.172726971 +0000 UTC m=+1444.534950522" watchObservedRunningTime="2026-09-16 00:58:14.175158193 +0000 UTC m=+1444.537381725" Sep 16 00:58:17.521354 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:17.521310 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b"] Sep 16 00:58:17.531888 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:17.531854 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:18.536915 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:18.536835 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:19.545307 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:19.545272 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:58:25.151310 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:25.151268 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s" Sep 16 00:58:25.152485 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:25.152449 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78" Sep 16 00:58:28.113586 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:28.113542 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:31.257934 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:31.257897 2567 generic.go:356] "Generic (PLEG): container finished" podID="6b00f081-7f0d-471b-bdd4-85bfe427477f" containerID="e12ee3610eb098c5b4beeee813475f8bcbf5112ade9ebb1c2227459c20ba8d71" exitCode=0 Sep 16 00:58:31.258400 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:31.257945 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" event={"ID":"6b00f081-7f0d-471b-bdd4-85bfe427477f","Type":"ContainerDied","Data":"e12ee3610eb098c5b4beeee813475f8bcbf5112ade9ebb1c2227459c20ba8d71"} Sep 16 00:58:31.258400 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:31.258390 2567 scope.go:117] "RemoveContainer" containerID="e12ee3610eb098c5b4beeee813475f8bcbf5112ade9ebb1c2227459c20ba8d71" Sep 16 00:58:31.259675 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:31.259649 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:58:31.259734 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:31.259724 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:58:31.259774 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:31.259755 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 00:58:32.272857 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.272814 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" event={"ID":"6b00f081-7f0d-471b-bdd4-85bfe427477f","Type":"ContainerStarted","Data":"d65d2dad31994911d4b9f5a2c69f6a6e552aaaae763e678c757c649379ac7715"} Sep 16 00:58:32.331880 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.331843 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:32.332314 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.332283 2567 patch_prober.go:28] interesting pod/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 16 00:58:32.332428 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.332385 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" podUID="6b00f081-7f0d-471b-bdd4-85bfe427477f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 16 00:58:32.542465 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.542351 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" podUID="abfb4e62-c537-4fb6-bc86-2a2ec0295113" containerName="main" containerID="cri-o://714a6a19569d1feb1676c0f3ca5120ccd932504b49f5a4fe164c8ec44d74b2c1" gracePeriod=45 Sep 16 00:58:32.826138 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.826108 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:58:32.872386 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.871391 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-dshm\") pod \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " Sep 16 00:58:32.872386 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.871440 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-home\") pod \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " Sep 16 00:58:32.872386 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.871501 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggkm6\" (UniqueName: \"kubernetes.io/projected/abfb4e62-c537-4fb6-bc86-2a2ec0295113-kube-api-access-ggkm6\") pod \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " Sep 16 00:58:32.872386 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.871541 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-tmp-dir\") pod \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " Sep 16 00:58:32.872386 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.871613 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-model-cache\") pod \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " Sep 16 00:58:32.872386 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.871660 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abfb4e62-c537-4fb6-bc86-2a2ec0295113-tls-certs\") pod \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\" (UID: \"abfb4e62-c537-4fb6-bc86-2a2ec0295113\") " Sep 16 00:58:32.872955 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.872864 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-home" (OuterVolumeSpecName: "home") pod "abfb4e62-c537-4fb6-bc86-2a2ec0295113" (UID: "abfb4e62-c537-4fb6-bc86-2a2ec0295113"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:58:32.873025 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.873004 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-model-cache" (OuterVolumeSpecName: "model-cache") pod "abfb4e62-c537-4fb6-bc86-2a2ec0295113" (UID: "abfb4e62-c537-4fb6-bc86-2a2ec0295113"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:58:32.873231 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.873203 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "abfb4e62-c537-4fb6-bc86-2a2ec0295113" (UID: "abfb4e62-c537-4fb6-bc86-2a2ec0295113"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:58:32.875244 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.875209 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-dshm" (OuterVolumeSpecName: "dshm") pod "abfb4e62-c537-4fb6-bc86-2a2ec0295113" (UID: "abfb4e62-c537-4fb6-bc86-2a2ec0295113"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 00:58:32.875363 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.875321 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abfb4e62-c537-4fb6-bc86-2a2ec0295113-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "abfb4e62-c537-4fb6-bc86-2a2ec0295113" (UID: "abfb4e62-c537-4fb6-bc86-2a2ec0295113"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 00:58:32.875405 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.875373 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfb4e62-c537-4fb6-bc86-2a2ec0295113-kube-api-access-ggkm6" (OuterVolumeSpecName: "kube-api-access-ggkm6") pod "abfb4e62-c537-4fb6-bc86-2a2ec0295113" (UID: "abfb4e62-c537-4fb6-bc86-2a2ec0295113"). InnerVolumeSpecName "kube-api-access-ggkm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 00:58:32.973222 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.973183 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggkm6\" (UniqueName: \"kubernetes.io/projected/abfb4e62-c537-4fb6-bc86-2a2ec0295113-kube-api-access-ggkm6\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:58:32.973222 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.973218 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-tmp-dir\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:58:32.973222 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.973232 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-model-cache\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:58:32.973480 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.973240 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abfb4e62-c537-4fb6-bc86-2a2ec0295113-tls-certs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:58:32.973480 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.973247 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-dshm\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:58:32.973480 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:32.973255 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abfb4e62-c537-4fb6-bc86-2a2ec0295113-home\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:58:33.280216 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:33.280118 2567 generic.go:356] "Generic (PLEG): container finished" podID="abfb4e62-c537-4fb6-bc86-2a2ec0295113" containerID="714a6a19569d1feb1676c0f3ca5120ccd932504b49f5a4fe164c8ec44d74b2c1" exitCode=0 Sep 16 00:58:33.280216 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:33.280168 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" Sep 16 00:58:33.280683 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:33.280209 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" event={"ID":"abfb4e62-c537-4fb6-bc86-2a2ec0295113","Type":"ContainerDied","Data":"714a6a19569d1feb1676c0f3ca5120ccd932504b49f5a4fe164c8ec44d74b2c1"} Sep 16 00:58:33.280683 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:33.280257 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b" event={"ID":"abfb4e62-c537-4fb6-bc86-2a2ec0295113","Type":"ContainerDied","Data":"cfa1803ce1f59d8519f5c2080ad3644a13cc50330220033845df6864ed4c3478"} Sep 16 00:58:33.280683 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:33.280275 2567 scope.go:117] "RemoveContainer" containerID="714a6a19569d1feb1676c0f3ca5120ccd932504b49f5a4fe164c8ec44d74b2c1" Sep 16 00:58:33.290910 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:33.290891 2567 scope.go:117] "RemoveContainer" containerID="714a6a19569d1feb1676c0f3ca5120ccd932504b49f5a4fe164c8ec44d74b2c1" Sep 16 00:58:33.291331 ip-10-0-131-194 kubenswrapper[2567]: E0916 00:58:33.291309 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"714a6a19569d1feb1676c0f3ca5120ccd932504b49f5a4fe164c8ec44d74b2c1\": container with ID starting with 714a6a19569d1feb1676c0f3ca5120ccd932504b49f5a4fe164c8ec44d74b2c1 not found: ID does not exist" containerID="714a6a19569d1feb1676c0f3ca5120ccd932504b49f5a4fe164c8ec44d74b2c1" Sep 16 00:58:33.291410 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:33.291341 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"714a6a19569d1feb1676c0f3ca5120ccd932504b49f5a4fe164c8ec44d74b2c1"} err="failed to get container status \"714a6a19569d1feb1676c0f3ca5120ccd932504b49f5a4fe164c8ec44d74b2c1\": rpc error: code = NotFound desc = could not find container \"714a6a19569d1feb1676c0f3ca5120ccd932504b49f5a4fe164c8ec44d74b2c1\": container with ID starting with 714a6a19569d1feb1676c0f3ca5120ccd932504b49f5a4fe164c8ec44d74b2c1 not found: ID does not exist" Sep 16 00:58:33.303602 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:33.303547 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b"] Sep 16 00:58:33.308904 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:33.308866 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-108960/sub-test-model-108960-kserve-5985dcc8dc-q8h7b"] Sep 16 00:58:33.332215 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:33.332175 2567 patch_prober.go:28] interesting pod/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 16 00:58:33.332398 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:33.332251 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" podUID="6b00f081-7f0d-471b-bdd4-85bfe427477f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 16 00:58:34.122547 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:34.122508 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abfb4e62-c537-4fb6-bc86-2a2ec0295113" path="/var/lib/kubelet/pods/abfb4e62-c537-4fb6-bc86-2a2ec0295113/volumes" Sep 16 00:58:34.332301 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:34.332263 2567 patch_prober.go:28] interesting pod/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 16 00:58:34.332715 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:34.332327 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" podUID="6b00f081-7f0d-471b-bdd4-85bfe427477f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 16 00:58:34.756482 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:34.756422 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:35.032017 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:35.031931 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:35.332249 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:35.332210 2567 patch_prober.go:28] interesting pod/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 16 00:58:35.332435 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:35.332280 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" podUID="6b00f081-7f0d-471b-bdd4-85bfe427477f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 16 00:58:35.685189 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:35.685072 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:36.332373 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:36.332325 2567 patch_prober.go:28] interesting pod/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 16 00:58:36.332579 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:36.332399 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" podUID="6b00f081-7f0d-471b-bdd4-85bfe427477f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 16 00:58:36.503746 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:36.503700 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:37.194023 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:37.193988 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:37.331945 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:37.331895 2567 patch_prober.go:28] interesting pod/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 16 00:58:37.332158 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:37.331956 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" podUID="6b00f081-7f0d-471b-bdd4-85bfe427477f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 16 00:58:38.043822 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:38.043784 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:58:38.337174 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:38.337133 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:38.337456 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:38.337435 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:38.338218 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:38.338200 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 00:58:40.543795 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:58:40.543750 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:59:00.454540 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:00.454506 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:59:10.310713 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:10.310676 2567 scope.go:117] "RemoveContainer" containerID="bd461d9b584f5c983822b52c525d696369d98d03a7b50caf893e5f9209348bfb" Sep 16 00:59:13.933755 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:13.933716 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:59:18.937742 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:18.937701 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:59:38.367291 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:38.367197 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:59:45.543150 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:45.543108 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37b46587-n5gtx"] Sep 16 00:59:45.543657 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:45.543533 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abfb4e62-c537-4fb6-bc86-2a2ec0295113" containerName="main" Sep 16 00:59:45.543657 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:45.543545 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="abfb4e62-c537-4fb6-bc86-2a2ec0295113" containerName="main" Sep 16 00:59:45.543657 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:45.543623 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="abfb4e62-c537-4fb6-bc86-2a2ec0295113" containerName="main" Sep 16 00:59:45.546739 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:45.546720 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37b46587-n5gtx" Sep 16 00:59:45.549735 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:45.549706 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 00:59:45.549903 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:45.549717 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-npltg\"" Sep 16 00:59:45.556387 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:45.556361 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37b46587-n5gtx"] Sep 16 00:59:45.622971 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:45.622932 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e39802c-6dfb-4962-8de9-35635eeab177-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-37b46587-n5gtx\" (UID: \"5e39802c-6dfb-4962-8de9-35635eeab177\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37b46587-n5gtx" Sep 16 00:59:45.724343 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:45.724310 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e39802c-6dfb-4962-8de9-35635eeab177-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-37b46587-n5gtx\" (UID: \"5e39802c-6dfb-4962-8de9-35635eeab177\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37b46587-n5gtx" Sep 16 00:59:45.724926 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:45.724906 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e39802c-6dfb-4962-8de9-35635eeab177-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-37b46587-n5gtx\" (UID: \"5e39802c-6dfb-4962-8de9-35635eeab177\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37b46587-n5gtx" Sep 16 00:59:45.858941 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:45.858901 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37b46587-n5gtx" Sep 16 00:59:46.138872 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:46.138844 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37b46587-n5gtx"] Sep 16 00:59:46.140104 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:59:46.140069 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e39802c_6dfb_4962_8de9_35635eeab177.slice/crio-7df05528a07c2f546e8d6f11934ed5cfc7a22b86b23c52788c0ccbe8b4327161 WatchSource:0}: Error finding container 7df05528a07c2f546e8d6f11934ed5cfc7a22b86b23c52788c0ccbe8b4327161: Status 404 returned error can't find the container with id 7df05528a07c2f546e8d6f11934ed5cfc7a22b86b23c52788c0ccbe8b4327161 Sep 16 00:59:46.540569 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:46.540474 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:59:46.734607 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:46.734504 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37b46587-n5gtx" event={"ID":"5e39802c-6dfb-4962-8de9-35635eeab177","Type":"ContainerStarted","Data":"b3c6fa549c5a0186beae1d7bfe0336be7c0ac37facfff110aef868d8d28e93d5"} Sep 16 00:59:46.734607 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:46.734562 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37b46587-n5gtx" event={"ID":"5e39802c-6dfb-4962-8de9-35635eeab177","Type":"ContainerStarted","Data":"7df05528a07c2f546e8d6f11934ed5cfc7a22b86b23c52788c0ccbe8b4327161"} Sep 16 00:59:46.756596 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:46.756535 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37b46587-n5gtx" podStartSLOduration=1.7565153119999999 podStartE2EDuration="1.756515312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:59:46.753904652 +0000 UTC m=+1537.116128212" watchObservedRunningTime="2026-09-16 00:59:46.756515312 +0000 UTC m=+1537.118738893" Sep 16 00:59:47.742509 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:47.742472 2567 generic.go:356] "Generic (PLEG): container finished" podID="5e39802c-6dfb-4962-8de9-35635eeab177" containerID="b3c6fa549c5a0186beae1d7bfe0336be7c0ac37facfff110aef868d8d28e93d5" exitCode=0 Sep 16 00:59:47.742896 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:47.742565 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37b46587-n5gtx" event={"ID":"5e39802c-6dfb-4962-8de9-35635eeab177","Type":"ContainerDied","Data":"b3c6fa549c5a0186beae1d7bfe0336be7c0ac37facfff110aef868d8d28e93d5"} Sep 16 00:59:47.899017 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:47.898983 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:59:48.149271 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:48.149233 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:59:48.893245 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:48.893215 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37b46587-n5gtx" Sep 16 00:59:48.957329 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:48.957294 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e39802c-6dfb-4962-8de9-35635eeab177-maas-api-service-ca\") pod \"5e39802c-6dfb-4962-8de9-35635eeab177\" (UID: \"5e39802c-6dfb-4962-8de9-35635eeab177\") " Sep 16 00:59:48.957716 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:48.957692 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e39802c-6dfb-4962-8de9-35635eeab177-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e39802c-6dfb-4962-8de9-35635eeab177" (UID: "5e39802c-6dfb-4962-8de9-35635eeab177"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 00:59:48.966237 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:48.966198 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:59:49.058387 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.058351 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e39802c-6dfb-4962-8de9-35635eeab177-maas-api-service-ca\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 00:59:49.417317 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.417217 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t"] Sep 16 00:59:49.417720 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.417703 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e39802c-6dfb-4962-8de9-35635eeab177" containerName="revoke-keys" Sep 16 00:59:49.417768 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.417724 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e39802c-6dfb-4962-8de9-35635eeab177" containerName="revoke-keys" Sep 16 00:59:49.417806 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.417790 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e39802c-6dfb-4962-8de9-35635eeab177" containerName="revoke-keys" Sep 16 00:59:49.421078 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.421032 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:49.426925 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.426890 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-2fa86d\"/\"openshift-service-ca.crt\"" Sep 16 00:59:49.427137 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.426951 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-2fa86d\"/\"default-dockercfg-rfc58\"" Sep 16 00:59:49.427137 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.426956 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-2fa86d\"/\"rate-test-model-2fa86d-kserve-self-signed-certs\"" Sep 16 00:59:49.427354 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.427338 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-2fa86d\"/\"kube-root-ca.crt\"" Sep 16 00:59:49.439406 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.439372 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t"] Sep 16 00:59:49.463142 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.463106 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-tmp-dir\") pod \"rate-test-model-2fa86d-kserve-c94745895-sml9t\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:49.463307 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.463160 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-model-cache\") pod \"rate-test-model-2fa86d-kserve-c94745895-sml9t\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:49.463307 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.463208 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-dshm\") pod \"rate-test-model-2fa86d-kserve-c94745895-sml9t\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:49.463307 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.463241 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b10712-64cc-453d-9a7f-534fc5d8789e-tls-certs\") pod \"rate-test-model-2fa86d-kserve-c94745895-sml9t\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:49.463307 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.463264 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv5h9\" (UniqueName: \"kubernetes.io/projected/76b10712-64cc-453d-9a7f-534fc5d8789e-kube-api-access-lv5h9\") pod \"rate-test-model-2fa86d-kserve-c94745895-sml9t\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:49.463466 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.463342 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-home\") pod \"rate-test-model-2fa86d-kserve-c94745895-sml9t\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:49.474851 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.474814 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:59:49.564371 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.564331 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-home\") pod \"rate-test-model-2fa86d-kserve-c94745895-sml9t\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:49.564830 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.564792 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-tmp-dir\") pod \"rate-test-model-2fa86d-kserve-c94745895-sml9t\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:49.565029 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.565013 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-model-cache\") pod \"rate-test-model-2fa86d-kserve-c94745895-sml9t\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:49.565213 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.565198 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-dshm\") pod \"rate-test-model-2fa86d-kserve-c94745895-sml9t\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:49.565814 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.565337 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-tmp-dir\") pod \"rate-test-model-2fa86d-kserve-c94745895-sml9t\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:49.565814 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.565348 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b10712-64cc-453d-9a7f-534fc5d8789e-tls-certs\") pod \"rate-test-model-2fa86d-kserve-c94745895-sml9t\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:49.565814 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.565088 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-home\") pod \"rate-test-model-2fa86d-kserve-c94745895-sml9t\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:49.565814 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.565403 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv5h9\" (UniqueName: \"kubernetes.io/projected/76b10712-64cc-453d-9a7f-534fc5d8789e-kube-api-access-lv5h9\") pod \"rate-test-model-2fa86d-kserve-c94745895-sml9t\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:49.565814 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.565470 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-model-cache\") pod \"rate-test-model-2fa86d-kserve-c94745895-sml9t\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:49.569212 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.569183 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b10712-64cc-453d-9a7f-534fc5d8789e-tls-certs\") pod \"rate-test-model-2fa86d-kserve-c94745895-sml9t\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:49.569688 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.569593 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-dshm\") pod \"rate-test-model-2fa86d-kserve-c94745895-sml9t\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:49.578423 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.578389 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv5h9\" (UniqueName: \"kubernetes.io/projected/76b10712-64cc-453d-9a7f-534fc5d8789e-kube-api-access-lv5h9\") pod \"rate-test-model-2fa86d-kserve-c94745895-sml9t\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:49.731934 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.731832 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:49.756619 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.756587 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37b46587-n5gtx" Sep 16 00:59:49.756619 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.756615 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37b46587-n5gtx" event={"ID":"5e39802c-6dfb-4962-8de9-35635eeab177","Type":"ContainerDied","Data":"7df05528a07c2f546e8d6f11934ed5cfc7a22b86b23c52788c0ccbe8b4327161"} Sep 16 00:59:49.756876 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.756662 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7df05528a07c2f546e8d6f11934ed5cfc7a22b86b23c52788c0ccbe8b4327161" Sep 16 00:59:49.915178 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:49.915143 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 00:59:50.028440 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:50.028410 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t"] Sep 16 00:59:50.030247 ip-10-0-131-194 kubenswrapper[2567]: W0916 00:59:50.030214 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76b10712_64cc_453d_9a7f_534fc5d8789e.slice/crio-0c7eb4b15f284de643ae5fc6fe872d97cd5955fbbb7c4db6e958785cc465d755 WatchSource:0}: Error finding container 0c7eb4b15f284de643ae5fc6fe872d97cd5955fbbb7c4db6e958785cc465d755: Status 404 returned error can't find the container with id 0c7eb4b15f284de643ae5fc6fe872d97cd5955fbbb7c4db6e958785cc465d755 Sep 16 00:59:50.764744 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:50.764701 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" event={"ID":"76b10712-64cc-453d-9a7f-534fc5d8789e","Type":"ContainerStarted","Data":"919b53d2a2a4b6390b2fd231b7194124ad55e72aa6a58a6ccb8cffb35c95dbb0"} Sep 16 00:59:50.764744 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:50.764743 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" event={"ID":"76b10712-64cc-453d-9a7f-534fc5d8789e","Type":"ContainerStarted","Data":"0c7eb4b15f284de643ae5fc6fe872d97cd5955fbbb7c4db6e958785cc465d755"} Sep 16 00:59:50.792985 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:50.792916 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" podStartSLOduration=1.792897242 podStartE2EDuration="1.792897242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:59:50.791624179 +0000 UTC m=+1541.153847732" watchObservedRunningTime="2026-09-16 00:59:50.792897242 +0000 UTC m=+1541.155120773" Sep 16 00:59:55.863193 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:55.863149 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 00:59:59.732836 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:59.732786 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:59.732836 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:59.732849 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:59.746118 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:59.746083 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 00:59:59.832026 ip-10-0-131-194 kubenswrapper[2567]: I0916 00:59:59.831984 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 01:00:00.131119 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.131062 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2fa86d-29825340-d284r"] Sep 16 01:00:00.135868 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.135833 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2fa86d-29825340-d284r" Sep 16 01:00:00.136075 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.135895 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3462ea-29825340-tvwrb"] Sep 16 01:00:00.138474 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.138450 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-npltg\"" Sep 16 01:00:00.139904 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.139885 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825340-rkztz"] Sep 16 01:00:00.140024 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.140000 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3462ea-29825340-tvwrb" Sep 16 01:00:00.143720 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.143699 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825340-rkztz" Sep 16 01:00:00.149403 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.149365 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2fa86d-29825340-d284r" Sep 16 01:00:00.150066 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.149998 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2fa86d-29825340-d284r"] Sep 16 01:00:00.154641 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.154591 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3462ea-29825340-tvwrb"] Sep 16 01:00:00.158599 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.158574 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825340-rkztz"] Sep 16 01:00:00.163344 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.163310 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3462ea-29825340-tvwrb" Sep 16 01:00:00.208603 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.208574 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825340-rkztz" Sep 16 01:00:00.447174 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.447142 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2fa86d-29825340-d284r"] Sep 16 01:00:00.449372 ip-10-0-131-194 kubenswrapper[2567]: W0916 01:00:00.449343 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba1d72e_99f5_4d36_994a_d806e28cd1bb.slice/crio-90cf382bdc44d2c76cdd685f576aea4d92ceaf4c2bebfaffdb4e53c99c8c0970 WatchSource:0}: Error finding container 90cf382bdc44d2c76cdd685f576aea4d92ceaf4c2bebfaffdb4e53c99c8c0970: Status 404 returned error can't find the container with id 90cf382bdc44d2c76cdd685f576aea4d92ceaf4c2bebfaffdb4e53c99c8c0970 Sep 16 01:00:00.466871 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.466842 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3462ea-29825340-tvwrb"] Sep 16 01:00:00.468783 ip-10-0-131-194 kubenswrapper[2567]: W0916 01:00:00.468747 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24325c82_c840_4ea5_8df5_dc873e840b57.slice/crio-f4e51fe69264ebf22671ecee683c2cf7769a0c35e408015e4351777350e6bc07 WatchSource:0}: Error finding container f4e51fe69264ebf22671ecee683c2cf7769a0c35e408015e4351777350e6bc07: Status 404 returned error can't find the container with id f4e51fe69264ebf22671ecee683c2cf7769a0c35e408015e4351777350e6bc07 Sep 16 01:00:00.516818 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.516644 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825340-rkztz"] Sep 16 01:00:00.519660 ip-10-0-131-194 kubenswrapper[2567]: W0916 01:00:00.519617 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08904bd_114b_40c0_8e28_f942bea96427.slice/crio-647d3912fe29104bb9b5b966f264bcf5baeade931c83217dadcf29b9bf06f1b2 WatchSource:0}: Error finding container 647d3912fe29104bb9b5b966f264bcf5baeade931c83217dadcf29b9bf06f1b2: Status 404 returned error can't find the container with id 647d3912fe29104bb9b5b966f264bcf5baeade931c83217dadcf29b9bf06f1b2 Sep 16 01:00:00.827954 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.827907 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2fa86d-29825340-d284r" event={"ID":"8ba1d72e-99f5-4d36-994a-d806e28cd1bb","Type":"ContainerStarted","Data":"c2022217ce2ddef537c3ae3175197674efe4df744cfe4a6b2ecc4c79bd642cd7"} Sep 16 01:00:00.828405 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.827978 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2fa86d-29825340-d284r" event={"ID":"8ba1d72e-99f5-4d36-994a-d806e28cd1bb","Type":"ContainerStarted","Data":"90cf382bdc44d2c76cdd685f576aea4d92ceaf4c2bebfaffdb4e53c99c8c0970"} Sep 16 01:00:00.829315 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.829289 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825340-rkztz" event={"ID":"f08904bd-114b-40c0-8e28-f942bea96427","Type":"ContainerStarted","Data":"647d3912fe29104bb9b5b966f264bcf5baeade931c83217dadcf29b9bf06f1b2"} Sep 16 01:00:00.830604 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.830578 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3462ea-29825340-tvwrb" event={"ID":"24325c82-c840-4ea5-8df5-dc873e840b57","Type":"ContainerStarted","Data":"f4e51fe69264ebf22671ecee683c2cf7769a0c35e408015e4351777350e6bc07"} Sep 16 01:00:00.848118 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:00.848055 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2fa86d-29825340-d284r" podStartSLOduration=0.848015797 podStartE2EDuration="848.015797ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:00:00.844838607 +0000 UTC m=+1551.207062166" watchObservedRunningTime="2026-09-16 01:00:00.848015797 +0000 UTC m=+1551.210239395" Sep 16 01:00:01.308295 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.308248 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78"] Sep 16 01:00:01.308818 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.308784 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78" podUID="cb4bf108-a2c2-40e0-a6da-d3ffa2368982" containerName="payload-processing" containerID="cri-o://4c2e60d94043c59e5c188957a6484ec62d900574740a99965bc401c741083780" gracePeriod=30 Sep 16 01:00:01.318429 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.318374 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s"] Sep 16 01:00:01.318979 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.318669 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s" podUID="07adb093-0a0b-4c76-b510-a5b54e8a3f19" containerName="payload-pre-processing" containerID="cri-o://58179725b2b4c56163d18f3cd0c70dc13f3858a44b52f54c3524d7dd34135bec" gracePeriod=30 Sep 16 01:00:01.335493 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.335427 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:01.470556 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.470522 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:01.507180 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.506494 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37b46587-n5gtx"] Sep 16 01:00:01.509790 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.509743 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37b46587-n5gtx"] Sep 16 01:00:01.619404 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.619370 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s" Sep 16 01:00:01.642630 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.642603 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78" Sep 16 01:00:01.689583 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.689549 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:01.699731 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.699702 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf74p\" (UniqueName: \"kubernetes.io/projected/07adb093-0a0b-4c76-b510-a5b54e8a3f19-kube-api-access-gf74p\") pod \"07adb093-0a0b-4c76-b510-a5b54e8a3f19\" (UID: \"07adb093-0a0b-4c76-b510-a5b54e8a3f19\") " Sep 16 01:00:01.699893 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.699800 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07adb093-0a0b-4c76-b510-a5b54e8a3f19-plugins-config-volume\") pod \"07adb093-0a0b-4c76-b510-a5b54e8a3f19\" (UID: \"07adb093-0a0b-4c76-b510-a5b54e8a3f19\") " Sep 16 01:00:01.700304 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.700281 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07adb093-0a0b-4c76-b510-a5b54e8a3f19-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "07adb093-0a0b-4c76-b510-a5b54e8a3f19" (UID: "07adb093-0a0b-4c76-b510-a5b54e8a3f19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:00:01.702176 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.702151 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07adb093-0a0b-4c76-b510-a5b54e8a3f19-kube-api-access-gf74p" (OuterVolumeSpecName: "kube-api-access-gf74p") pod "07adb093-0a0b-4c76-b510-a5b54e8a3f19" (UID: "07adb093-0a0b-4c76-b510-a5b54e8a3f19"). InnerVolumeSpecName "kube-api-access-gf74p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:00:01.800818 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.800771 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4bf108-a2c2-40e0-a6da-d3ffa2368982-plugins-config-volume\") pod \"cb4bf108-a2c2-40e0-a6da-d3ffa2368982\" (UID: \"cb4bf108-a2c2-40e0-a6da-d3ffa2368982\") " Sep 16 01:00:01.800818 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.800814 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7szdv\" (UniqueName: \"kubernetes.io/projected/cb4bf108-a2c2-40e0-a6da-d3ffa2368982-kube-api-access-7szdv\") pod \"cb4bf108-a2c2-40e0-a6da-d3ffa2368982\" (UID: \"cb4bf108-a2c2-40e0-a6da-d3ffa2368982\") " Sep 16 01:00:01.801092 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.801071 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf74p\" (UniqueName: \"kubernetes.io/projected/07adb093-0a0b-4c76-b510-a5b54e8a3f19-kube-api-access-gf74p\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:00:01.801092 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.801089 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07adb093-0a0b-4c76-b510-a5b54e8a3f19-plugins-config-volume\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:00:01.801284 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.801258 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb4bf108-a2c2-40e0-a6da-d3ffa2368982-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb4bf108-a2c2-40e0-a6da-d3ffa2368982" (UID: "cb4bf108-a2c2-40e0-a6da-d3ffa2368982"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:00:01.803357 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.803323 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb4bf108-a2c2-40e0-a6da-d3ffa2368982-kube-api-access-7szdv" (OuterVolumeSpecName: "kube-api-access-7szdv") pod "cb4bf108-a2c2-40e0-a6da-d3ffa2368982" (UID: "cb4bf108-a2c2-40e0-a6da-d3ffa2368982"). InnerVolumeSpecName "kube-api-access-7szdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:00:01.838644 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.838549 2567 generic.go:356] "Generic (PLEG): container finished" podID="07adb093-0a0b-4c76-b510-a5b54e8a3f19" containerID="58179725b2b4c56163d18f3cd0c70dc13f3858a44b52f54c3524d7dd34135bec" exitCode=0 Sep 16 01:00:01.838644 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.838596 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s" Sep 16 01:00:01.838644 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.838632 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s" event={"ID":"07adb093-0a0b-4c76-b510-a5b54e8a3f19","Type":"ContainerDied","Data":"58179725b2b4c56163d18f3cd0c70dc13f3858a44b52f54c3524d7dd34135bec"} Sep 16 01:00:01.839235 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.838689 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s" event={"ID":"07adb093-0a0b-4c76-b510-a5b54e8a3f19","Type":"ContainerDied","Data":"e8c4b840a702787e98f31b62e5d6e9bc8f6df295439999f3b73e22f182a062f4"} Sep 16 01:00:01.839235 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.838713 2567 scope.go:117] "RemoveContainer" containerID="58179725b2b4c56163d18f3cd0c70dc13f3858a44b52f54c3524d7dd34135bec" Sep 16 01:00:01.840562 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.840536 2567 generic.go:356] "Generic (PLEG): container finished" podID="8ba1d72e-99f5-4d36-994a-d806e28cd1bb" containerID="c2022217ce2ddef537c3ae3175197674efe4df744cfe4a6b2ecc4c79bd642cd7" exitCode=0 Sep 16 01:00:01.840696 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.840621 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2fa86d-29825340-d284r" event={"ID":"8ba1d72e-99f5-4d36-994a-d806e28cd1bb","Type":"ContainerDied","Data":"c2022217ce2ddef537c3ae3175197674efe4df744cfe4a6b2ecc4c79bd642cd7"} Sep 16 01:00:01.842160 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.842136 2567 generic.go:356] "Generic (PLEG): container finished" podID="cb4bf108-a2c2-40e0-a6da-d3ffa2368982" containerID="4c2e60d94043c59e5c188957a6484ec62d900574740a99965bc401c741083780" exitCode=0 Sep 16 01:00:01.842294 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.842198 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78" Sep 16 01:00:01.842294 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.842222 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78" event={"ID":"cb4bf108-a2c2-40e0-a6da-d3ffa2368982","Type":"ContainerDied","Data":"4c2e60d94043c59e5c188957a6484ec62d900574740a99965bc401c741083780"} Sep 16 01:00:01.842294 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.842276 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78" event={"ID":"cb4bf108-a2c2-40e0-a6da-d3ffa2368982","Type":"ContainerDied","Data":"c66b711d1d97c58b3a5aa4dfdefba2c3c41ca27529d8112596c6cd65edd1ecd7"} Sep 16 01:00:01.843982 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.843954 2567 generic.go:356] "Generic (PLEG): container finished" podID="f08904bd-114b-40c0-8e28-f942bea96427" containerID="e340cfd91ac4facf1618e666d14d53aac0135ac4cd03255daa6cae0811542f6d" exitCode=0 Sep 16 01:00:01.844148 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.844062 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825340-rkztz" event={"ID":"f08904bd-114b-40c0-8e28-f942bea96427","Type":"ContainerDied","Data":"e340cfd91ac4facf1618e666d14d53aac0135ac4cd03255daa6cae0811542f6d"} Sep 16 01:00:01.847995 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.847965 2567 generic.go:356] "Generic (PLEG): container finished" podID="24325c82-c840-4ea5-8df5-dc873e840b57" containerID="8d984acaaf88b0bc439c34ef0a27b7d4871d42546c683a44cff4ec2703634586" exitCode=0 Sep 16 01:00:01.848206 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.848024 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3462ea-29825340-tvwrb" event={"ID":"24325c82-c840-4ea5-8df5-dc873e840b57","Type":"ContainerDied","Data":"8d984acaaf88b0bc439c34ef0a27b7d4871d42546c683a44cff4ec2703634586"} Sep 16 01:00:01.851459 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.851436 2567 scope.go:117] "RemoveContainer" containerID="58179725b2b4c56163d18f3cd0c70dc13f3858a44b52f54c3524d7dd34135bec" Sep 16 01:00:01.851846 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:00:01.851824 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58179725b2b4c56163d18f3cd0c70dc13f3858a44b52f54c3524d7dd34135bec\": container with ID starting with 58179725b2b4c56163d18f3cd0c70dc13f3858a44b52f54c3524d7dd34135bec not found: ID does not exist" containerID="58179725b2b4c56163d18f3cd0c70dc13f3858a44b52f54c3524d7dd34135bec" Sep 16 01:00:01.851936 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.851860 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58179725b2b4c56163d18f3cd0c70dc13f3858a44b52f54c3524d7dd34135bec"} err="failed to get container status \"58179725b2b4c56163d18f3cd0c70dc13f3858a44b52f54c3524d7dd34135bec\": rpc error: code = NotFound desc = could not find container \"58179725b2b4c56163d18f3cd0c70dc13f3858a44b52f54c3524d7dd34135bec\": container with ID starting with 58179725b2b4c56163d18f3cd0c70dc13f3858a44b52f54c3524d7dd34135bec not found: ID does not exist" Sep 16 01:00:01.851936 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.851887 2567 scope.go:117] "RemoveContainer" containerID="4c2e60d94043c59e5c188957a6484ec62d900574740a99965bc401c741083780" Sep 16 01:00:01.862174 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.862152 2567 scope.go:117] "RemoveContainer" containerID="4c2e60d94043c59e5c188957a6484ec62d900574740a99965bc401c741083780" Sep 16 01:00:01.862532 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:00:01.862511 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2e60d94043c59e5c188957a6484ec62d900574740a99965bc401c741083780\": container with ID starting with 4c2e60d94043c59e5c188957a6484ec62d900574740a99965bc401c741083780 not found: ID does not exist" containerID="4c2e60d94043c59e5c188957a6484ec62d900574740a99965bc401c741083780" Sep 16 01:00:01.862619 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.862541 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2e60d94043c59e5c188957a6484ec62d900574740a99965bc401c741083780"} err="failed to get container status \"4c2e60d94043c59e5c188957a6484ec62d900574740a99965bc401c741083780\": rpc error: code = NotFound desc = could not find container \"4c2e60d94043c59e5c188957a6484ec62d900574740a99965bc401c741083780\": container with ID starting with 4c2e60d94043c59e5c188957a6484ec62d900574740a99965bc401c741083780 not found: ID does not exist" Sep 16 01:00:01.887457 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.887296 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s"] Sep 16 01:00:01.890233 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.890204 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-37b46587-58fdbbf5b6-zrs9s"] Sep 16 01:00:01.902228 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.902184 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4bf108-a2c2-40e0-a6da-d3ffa2368982-plugins-config-volume\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:00:01.902228 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.902225 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7szdv\" (UniqueName: \"kubernetes.io/projected/cb4bf108-a2c2-40e0-a6da-d3ffa2368982-kube-api-access-7szdv\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:00:01.918464 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.918418 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78"] Sep 16 01:00:01.924855 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.924820 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-37b46587-686b8c9db9-hwg78"] Sep 16 01:00:01.997763 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:01.997727 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:02.121098 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:02.120972 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07adb093-0a0b-4c76-b510-a5b54e8a3f19" path="/var/lib/kubelet/pods/07adb093-0a0b-4c76-b510-a5b54e8a3f19/volumes" Sep 16 01:00:02.121407 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:02.121392 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e39802c-6dfb-4962-8de9-35635eeab177" path="/var/lib/kubelet/pods/5e39802c-6dfb-4962-8de9-35635eeab177/volumes" Sep 16 01:00:02.121713 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:02.121701 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb4bf108-a2c2-40e0-a6da-d3ffa2368982" path="/var/lib/kubelet/pods/cb4bf108-a2c2-40e0-a6da-d3ffa2368982/volumes" Sep 16 01:00:02.753700 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:02.753672 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:03.011978 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:03.011901 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3462ea-29825340-tvwrb" Sep 16 01:00:03.055657 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:03.055631 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825340-rkztz" Sep 16 01:00:03.059185 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:03.059162 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2fa86d-29825340-d284r" Sep 16 01:00:03.595499 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:03.595465 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:03.866970 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:03.866849 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2fa86d-29825340-d284r" event={"ID":"8ba1d72e-99f5-4d36-994a-d806e28cd1bb","Type":"ContainerDied","Data":"90cf382bdc44d2c76cdd685f576aea4d92ceaf4c2bebfaffdb4e53c99c8c0970"} Sep 16 01:00:03.866970 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:03.866955 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90cf382bdc44d2c76cdd685f576aea4d92ceaf4c2bebfaffdb4e53c99c8c0970" Sep 16 01:00:03.867282 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:03.866866 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2fa86d-29825340-d284r" Sep 16 01:00:03.869214 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:03.869181 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825340-rkztz" Sep 16 01:00:03.869214 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:03.869190 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825340-rkztz" event={"ID":"f08904bd-114b-40c0-8e28-f942bea96427","Type":"ContainerDied","Data":"647d3912fe29104bb9b5b966f264bcf5baeade931c83217dadcf29b9bf06f1b2"} Sep 16 01:00:03.869448 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:03.869243 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="647d3912fe29104bb9b5b966f264bcf5baeade931c83217dadcf29b9bf06f1b2" Sep 16 01:00:03.870769 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:03.870741 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3462ea-29825340-tvwrb" Sep 16 01:00:03.870769 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:03.870757 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3462ea-29825340-tvwrb" event={"ID":"24325c82-c840-4ea5-8df5-dc873e840b57","Type":"ContainerDied","Data":"f4e51fe69264ebf22671ecee683c2cf7769a0c35e408015e4351777350e6bc07"} Sep 16 01:00:03.870951 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:03.870802 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4e51fe69264ebf22671ecee683c2cf7769a0c35e408015e4351777350e6bc07" Sep 16 01:00:04.317581 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:04.317536 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:04.790861 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:04.790748 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:05.511025 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:05.510973 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:05.907543 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:05.907505 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:10.342427 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:10.342393 2567 scope.go:117] "RemoveContainer" containerID="dea51509032f601a4958898acd46f680633a443bc18c106e50bc4cefa8a63ebf" Sep 16 01:00:10.352154 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:10.352121 2567 scope.go:117] "RemoveContainer" containerID="911d7ef5534e5a0b3c9fe01dafcc403541c8d20a5de9ab306cbeff63541573a3" Sep 16 01:00:10.361545 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:10.361523 2567 scope.go:117] "RemoveContainer" containerID="05e339fa736a5f00b175d89aa02dacde3c9ebe57028cc6517d797dce9121aafd" Sep 16 01:00:10.512968 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:10.512930 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:10.645084 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:10.644952 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:15.862356 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:15.862313 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:00:28.551271 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:28.551232 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:00:43.639969 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:43.639908 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:00:46.082402 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:46.082291 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:47.037763 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:47.037725 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:48.171579 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:48.171527 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:49.117265 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:49.117219 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t"] Sep 16 01:00:49.145472 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:49.145442 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:49.590825 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:49.590793 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:49.908162 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:49.908072 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:50.150510 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:50.150473 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:52.944427 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:52.944395 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:53.053773 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:53.053743 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:53.185867 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:53.185828 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn"] Sep 16 01:00:53.186108 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:53.186073 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" podUID="6b00f081-7f0d-471b-bdd4-85bfe427477f" containerName="istio-proxy" containerID="cri-o://d65d2dad31994911d4b9f5a2c69f6a6e552aaaae763e678c757c649379ac7715" gracePeriod=30 Sep 16 01:00:53.212116 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:53.211979 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:53.340031 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:53.339988 2567 patch_prober.go:28] interesting pod/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 01:00:53.340248 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:53.340110 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" podUID="6b00f081-7f0d-471b-bdd4-85bfe427477f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 01:00:53.438191 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:53.438143 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:53.535376 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:53.535282 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:53.652248 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:53.652214 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:54.542875 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:54.542827 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:54.645321 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:54.645267 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:54.743645 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:54.743590 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:54.844331 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:54.844294 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:55.408285 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:55.408247 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:56.871856 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:56.871810 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:56.969348 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:56.969312 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:57.051976 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.051945 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:57.120567 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.120539 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:57.327351 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.327317 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:57.531174 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.531134 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:57.626744 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.626652 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:00:57.667861 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.667819 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55"] Sep 16 01:00:57.668251 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.668238 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24325c82-c840-4ea5-8df5-dc873e840b57" containerName="cleanup" Sep 16 01:00:57.668301 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.668253 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="24325c82-c840-4ea5-8df5-dc873e840b57" containerName="cleanup" Sep 16 01:00:57.668301 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.668266 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb4bf108-a2c2-40e0-a6da-d3ffa2368982" containerName="payload-processing" Sep 16 01:00:57.668301 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.668272 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb4bf108-a2c2-40e0-a6da-d3ffa2368982" containerName="payload-processing" Sep 16 01:00:57.668301 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.668279 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba1d72e-99f5-4d36-994a-d806e28cd1bb" containerName="cleanup" Sep 16 01:00:57.668301 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.668286 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba1d72e-99f5-4d36-994a-d806e28cd1bb" containerName="cleanup" Sep 16 01:00:57.668301 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.668298 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f08904bd-114b-40c0-8e28-f942bea96427" containerName="cleanup" Sep 16 01:00:57.668301 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.668304 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f08904bd-114b-40c0-8e28-f942bea96427" containerName="cleanup" Sep 16 01:00:57.668526 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.668319 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07adb093-0a0b-4c76-b510-a5b54e8a3f19" containerName="payload-pre-processing" Sep 16 01:00:57.668526 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.668324 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="07adb093-0a0b-4c76-b510-a5b54e8a3f19" containerName="payload-pre-processing" Sep 16 01:00:57.668526 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.668384 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="07adb093-0a0b-4c76-b510-a5b54e8a3f19" containerName="payload-pre-processing" Sep 16 01:00:57.668526 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.668392 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="24325c82-c840-4ea5-8df5-dc873e840b57" containerName="cleanup" Sep 16 01:00:57.668526 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.668397 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f08904bd-114b-40c0-8e28-f942bea96427" containerName="cleanup" Sep 16 01:00:57.668526 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.668403 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba1d72e-99f5-4d36-994a-d806e28cd1bb" containerName="cleanup" Sep 16 01:00:57.668526 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.668410 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb4bf108-a2c2-40e0-a6da-d3ffa2368982" containerName="payload-processing" Sep 16 01:00:57.671802 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.671777 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55" Sep 16 01:00:57.675979 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.675957 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-715834\"" Sep 16 01:00:57.676140 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.675964 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-715834-dockercfg-lhkvw\"" Sep 16 01:00:57.689500 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.689468 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55"] Sep 16 01:00:57.773081 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.772985 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21cb4d08-16e2-4d5a-9f9e-59c394282a26-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55\" (UID: \"21cb4d08-16e2-4d5a-9f9e-59c394282a26\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55" Sep 16 01:00:57.773081 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.773079 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg9jp\" (UniqueName: \"kubernetes.io/projected/21cb4d08-16e2-4d5a-9f9e-59c394282a26-kube-api-access-tg9jp\") pod \"payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55\" (UID: \"21cb4d08-16e2-4d5a-9f9e-59c394282a26\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55" Sep 16 01:00:57.874581 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.874522 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21cb4d08-16e2-4d5a-9f9e-59c394282a26-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55\" (UID: \"21cb4d08-16e2-4d5a-9f9e-59c394282a26\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55" Sep 16 01:00:57.874581 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.874591 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg9jp\" (UniqueName: \"kubernetes.io/projected/21cb4d08-16e2-4d5a-9f9e-59c394282a26-kube-api-access-tg9jp\") pod \"payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55\" (UID: \"21cb4d08-16e2-4d5a-9f9e-59c394282a26\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55" Sep 16 01:00:57.875326 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.875305 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21cb4d08-16e2-4d5a-9f9e-59c394282a26-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55\" (UID: \"21cb4d08-16e2-4d5a-9f9e-59c394282a26\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55" Sep 16 01:00:57.884819 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.884747 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg9jp\" (UniqueName: \"kubernetes.io/projected/21cb4d08-16e2-4d5a-9f9e-59c394282a26-kube-api-access-tg9jp\") pod \"payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55\" (UID: \"21cb4d08-16e2-4d5a-9f9e-59c394282a26\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55" Sep 16 01:00:57.982606 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:57.982563 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55" Sep 16 01:00:58.267026 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.267000 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55"] Sep 16 01:00:58.274358 ip-10-0-131-194 kubenswrapper[2567]: W0916 01:00:58.274315 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21cb4d08_16e2_4d5a_9f9e_59c394282a26.slice/crio-545ec4b8dc338d278ddb05f89c1e833fa2359c30b5a22a01e24636e1252d339b WatchSource:0}: Error finding container 545ec4b8dc338d278ddb05f89c1e833fa2359c30b5a22a01e24636e1252d339b: Status 404 returned error can't find the container with id 545ec4b8dc338d278ddb05f89c1e833fa2359c30b5a22a01e24636e1252d339b Sep 16 01:00:58.454913 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.454878 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 01:00:58.582631 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.582598 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-data\") pod \"6b00f081-7f0d-471b-bdd4-85bfe427477f\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " Sep 16 01:00:58.582631 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.582639 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b00f081-7f0d-471b-bdd4-85bfe427477f-istiod-ca-cert\") pod \"6b00f081-7f0d-471b-bdd4-85bfe427477f\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " Sep 16 01:00:58.582935 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.582673 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpxrt\" (UniqueName: \"kubernetes.io/projected/6b00f081-7f0d-471b-bdd4-85bfe427477f-kube-api-access-wpxrt\") pod \"6b00f081-7f0d-471b-bdd4-85bfe427477f\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " Sep 16 01:00:58.582935 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.582702 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-token\") pod \"6b00f081-7f0d-471b-bdd4-85bfe427477f\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " Sep 16 01:00:58.583059 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.582945 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-data" (OuterVolumeSpecName: "istio-data") pod "6b00f081-7f0d-471b-bdd4-85bfe427477f" (UID: "6b00f081-7f0d-471b-bdd4-85bfe427477f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:00:58.583059 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.582945 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-envoy\") pod \"6b00f081-7f0d-471b-bdd4-85bfe427477f\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " Sep 16 01:00:58.583169 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.583093 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b00f081-7f0d-471b-bdd4-85bfe427477f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6b00f081-7f0d-471b-bdd4-85bfe427477f" (UID: "6b00f081-7f0d-471b-bdd4-85bfe427477f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:00:58.583169 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.583103 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-workload-certs\") pod \"6b00f081-7f0d-471b-bdd4-85bfe427477f\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " Sep 16 01:00:58.583169 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.583133 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-credential-socket\") pod \"6b00f081-7f0d-471b-bdd4-85bfe427477f\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " Sep 16 01:00:58.583325 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.583176 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-podinfo\") pod \"6b00f081-7f0d-471b-bdd4-85bfe427477f\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " Sep 16 01:00:58.583325 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.583209 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-workload-socket\") pod \"6b00f081-7f0d-471b-bdd4-85bfe427477f\" (UID: \"6b00f081-7f0d-471b-bdd4-85bfe427477f\") " Sep 16 01:00:58.583428 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.583385 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6b00f081-7f0d-471b-bdd4-85bfe427477f" (UID: "6b00f081-7f0d-471b-bdd4-85bfe427477f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:00:58.583517 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.583473 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6b00f081-7f0d-471b-bdd4-85bfe427477f" (UID: "6b00f081-7f0d-471b-bdd4-85bfe427477f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:00:58.583574 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.583549 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6b00f081-7f0d-471b-bdd4-85bfe427477f" (UID: "6b00f081-7f0d-471b-bdd4-85bfe427477f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:00:58.583676 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.583659 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-workload-certs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:00:58.583732 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.583684 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-credential-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:00:58.583732 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.583700 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-workload-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:00:58.583732 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.583714 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-data\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:00:58.583732 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.583727 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b00f081-7f0d-471b-bdd4-85bfe427477f-istiod-ca-cert\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:00:58.585332 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.585303 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6b00f081-7f0d-471b-bdd4-85bfe427477f" (UID: "6b00f081-7f0d-471b-bdd4-85bfe427477f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:00:58.585577 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.585554 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6b00f081-7f0d-471b-bdd4-85bfe427477f" (UID: "6b00f081-7f0d-471b-bdd4-85bfe427477f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 01:00:58.585743 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.585717 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-token" (OuterVolumeSpecName: "istio-token") pod "6b00f081-7f0d-471b-bdd4-85bfe427477f" (UID: "6b00f081-7f0d-471b-bdd4-85bfe427477f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:00:58.585844 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.585717 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b00f081-7f0d-471b-bdd4-85bfe427477f-kube-api-access-wpxrt" (OuterVolumeSpecName: "kube-api-access-wpxrt") pod "6b00f081-7f0d-471b-bdd4-85bfe427477f" (UID: "6b00f081-7f0d-471b-bdd4-85bfe427477f"). InnerVolumeSpecName "kube-api-access-wpxrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:00:58.684621 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.684577 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-podinfo\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:00:58.684621 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.684610 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpxrt\" (UniqueName: \"kubernetes.io/projected/6b00f081-7f0d-471b-bdd4-85bfe427477f-kube-api-access-wpxrt\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:00:58.684621 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.684621 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-token\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:00:58.684621 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:58.684630 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b00f081-7f0d-471b-bdd4-85bfe427477f-istio-envoy\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:00:59.218528 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:59.218423 2567 generic.go:356] "Generic (PLEG): container finished" podID="6b00f081-7f0d-471b-bdd4-85bfe427477f" containerID="d65d2dad31994911d4b9f5a2c69f6a6e552aaaae763e678c757c649379ac7715" exitCode=0 Sep 16 01:00:59.218528 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:59.218480 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" Sep 16 01:00:59.218528 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:59.218517 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" event={"ID":"6b00f081-7f0d-471b-bdd4-85bfe427477f","Type":"ContainerDied","Data":"d65d2dad31994911d4b9f5a2c69f6a6e552aaaae763e678c757c649379ac7715"} Sep 16 01:00:59.219156 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:59.218572 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn" event={"ID":"6b00f081-7f0d-471b-bdd4-85bfe427477f","Type":"ContainerDied","Data":"518eb0c872c574e238f88ee6235843e4bb11278af78f78e6f4e9aab3903b2a97"} Sep 16 01:00:59.219156 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:59.218598 2567 scope.go:117] "RemoveContainer" containerID="d65d2dad31994911d4b9f5a2c69f6a6e552aaaae763e678c757c649379ac7715" Sep 16 01:00:59.220696 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:59.220669 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55" event={"ID":"21cb4d08-16e2-4d5a-9f9e-59c394282a26","Type":"ContainerStarted","Data":"86ab8533f4e80e6ebcb4d9e7eeec2d855b1c86c96a319c02a0fca753d0731c82"} Sep 16 01:00:59.220808 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:59.220726 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55" event={"ID":"21cb4d08-16e2-4d5a-9f9e-59c394282a26","Type":"ContainerStarted","Data":"545ec4b8dc338d278ddb05f89c1e833fa2359c30b5a22a01e24636e1252d339b"} Sep 16 01:00:59.220863 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:59.220803 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55" Sep 16 01:00:59.229624 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:59.229602 2567 scope.go:117] "RemoveContainer" containerID="e12ee3610eb098c5b4beeee813475f8bcbf5112ade9ebb1c2227459c20ba8d71" Sep 16 01:00:59.239301 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:59.239275 2567 scope.go:117] "RemoveContainer" containerID="d65d2dad31994911d4b9f5a2c69f6a6e552aaaae763e678c757c649379ac7715" Sep 16 01:00:59.239670 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:00:59.239642 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d65d2dad31994911d4b9f5a2c69f6a6e552aaaae763e678c757c649379ac7715\": container with ID starting with d65d2dad31994911d4b9f5a2c69f6a6e552aaaae763e678c757c649379ac7715 not found: ID does not exist" containerID="d65d2dad31994911d4b9f5a2c69f6a6e552aaaae763e678c757c649379ac7715" Sep 16 01:00:59.239747 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:59.239684 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d65d2dad31994911d4b9f5a2c69f6a6e552aaaae763e678c757c649379ac7715"} err="failed to get container status \"d65d2dad31994911d4b9f5a2c69f6a6e552aaaae763e678c757c649379ac7715\": rpc error: code = NotFound desc = could not find container \"d65d2dad31994911d4b9f5a2c69f6a6e552aaaae763e678c757c649379ac7715\": container with ID starting with d65d2dad31994911d4b9f5a2c69f6a6e552aaaae763e678c757c649379ac7715 not found: ID does not exist" Sep 16 01:00:59.239747 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:59.239706 2567 scope.go:117] "RemoveContainer" containerID="e12ee3610eb098c5b4beeee813475f8bcbf5112ade9ebb1c2227459c20ba8d71" Sep 16 01:00:59.240482 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:00:59.240451 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e12ee3610eb098c5b4beeee813475f8bcbf5112ade9ebb1c2227459c20ba8d71\": container with ID starting with e12ee3610eb098c5b4beeee813475f8bcbf5112ade9ebb1c2227459c20ba8d71 not found: ID does not exist" containerID="e12ee3610eb098c5b4beeee813475f8bcbf5112ade9ebb1c2227459c20ba8d71" Sep 16 01:00:59.240609 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:59.240490 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e12ee3610eb098c5b4beeee813475f8bcbf5112ade9ebb1c2227459c20ba8d71"} err="failed to get container status \"e12ee3610eb098c5b4beeee813475f8bcbf5112ade9ebb1c2227459c20ba8d71\": rpc error: code = NotFound desc = could not find container \"e12ee3610eb098c5b4beeee813475f8bcbf5112ade9ebb1c2227459c20ba8d71\": container with ID starting with e12ee3610eb098c5b4beeee813475f8bcbf5112ade9ebb1c2227459c20ba8d71 not found: ID does not exist" Sep 16 01:00:59.242550 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:59.242505 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55" podStartSLOduration=2.242490443 podStartE2EDuration="2.242490443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:00:59.24045126 +0000 UTC m=+1609.602674813" watchObservedRunningTime="2026-09-16 01:00:59.242490443 +0000 UTC m=+1609.604713975" Sep 16 01:00:59.255595 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:59.255554 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn"] Sep 16 01:00:59.259076 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:00:59.259029 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-37b46587-openshift-default-6447c4fcb8-x66kn"] Sep 16 01:01:00.119794 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:00.119755 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b00f081-7f0d-471b-bdd4-85bfe427477f" path="/var/lib/kubelet/pods/6b00f081-7f0d-471b-bdd4-85bfe427477f/volumes" Sep 16 01:01:00.845116 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:00.845072 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:01:03.415530 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:03.415486 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:03.515783 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:03.515747 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:04.139925 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:04.139881 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" podUID="76b10712-64cc-453d-9a7f-534fc5d8789e" containerName="main" containerID="cri-o://919b53d2a2a4b6390b2fd231b7194124ad55e72aa6a58a6ccb8cffb35c95dbb0" gracePeriod=45 Sep 16 01:01:04.403090 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:04.403006 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 01:01:04.549049 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:04.549010 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-dshm\") pod \"76b10712-64cc-453d-9a7f-534fc5d8789e\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " Sep 16 01:01:04.549540 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:04.549105 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-home\") pod \"76b10712-64cc-453d-9a7f-534fc5d8789e\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " Sep 16 01:01:04.549540 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:04.549130 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-model-cache\") pod \"76b10712-64cc-453d-9a7f-534fc5d8789e\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " Sep 16 01:01:04.549540 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:04.549165 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-tmp-dir\") pod \"76b10712-64cc-453d-9a7f-534fc5d8789e\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " Sep 16 01:01:04.549540 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:04.549183 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lv5h9\" (UniqueName: \"kubernetes.io/projected/76b10712-64cc-453d-9a7f-534fc5d8789e-kube-api-access-lv5h9\") pod \"76b10712-64cc-453d-9a7f-534fc5d8789e\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " Sep 16 01:01:04.549540 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:04.549256 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b10712-64cc-453d-9a7f-534fc5d8789e-tls-certs\") pod \"76b10712-64cc-453d-9a7f-534fc5d8789e\" (UID: \"76b10712-64cc-453d-9a7f-534fc5d8789e\") " Sep 16 01:01:04.549540 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:04.549391 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76b10712-64cc-453d-9a7f-534fc5d8789e" (UID: "76b10712-64cc-453d-9a7f-534fc5d8789e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:01:04.549540 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:04.549415 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-model-cache" (OuterVolumeSpecName: "model-cache") pod "76b10712-64cc-453d-9a7f-534fc5d8789e" (UID: "76b10712-64cc-453d-9a7f-534fc5d8789e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:01:04.549540 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:04.549438 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-home" (OuterVolumeSpecName: "home") pod "76b10712-64cc-453d-9a7f-534fc5d8789e" (UID: "76b10712-64cc-453d-9a7f-534fc5d8789e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:01:04.549935 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:04.549588 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-home\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:01:04.549935 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:04.549608 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-model-cache\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:01:04.549935 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:04.549626 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-tmp-dir\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:01:04.551511 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:04.551488 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-dshm" (OuterVolumeSpecName: "dshm") pod "76b10712-64cc-453d-9a7f-534fc5d8789e" (UID: "76b10712-64cc-453d-9a7f-534fc5d8789e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:01:04.551611 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:04.551548 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76b10712-64cc-453d-9a7f-534fc5d8789e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76b10712-64cc-453d-9a7f-534fc5d8789e" (UID: "76b10712-64cc-453d-9a7f-534fc5d8789e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:01:04.551650 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:04.551616 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b10712-64cc-453d-9a7f-534fc5d8789e-kube-api-access-lv5h9" (OuterVolumeSpecName: "kube-api-access-lv5h9") pod "76b10712-64cc-453d-9a7f-534fc5d8789e" (UID: "76b10712-64cc-453d-9a7f-534fc5d8789e"). InnerVolumeSpecName "kube-api-access-lv5h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:01:04.651064 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:04.651001 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b10712-64cc-453d-9a7f-534fc5d8789e-dshm\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:01:04.651265 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:04.651078 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv5h9\" (UniqueName: \"kubernetes.io/projected/76b10712-64cc-453d-9a7f-534fc5d8789e-kube-api-access-lv5h9\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:01:04.651265 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:04.651095 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b10712-64cc-453d-9a7f-534fc5d8789e-tls-certs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:01:05.261697 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:05.261660 2567 generic.go:356] "Generic (PLEG): container finished" podID="76b10712-64cc-453d-9a7f-534fc5d8789e" containerID="919b53d2a2a4b6390b2fd231b7194124ad55e72aa6a58a6ccb8cffb35c95dbb0" exitCode=0 Sep 16 01:01:05.261919 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:05.261714 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" Sep 16 01:01:05.261919 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:05.261744 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" event={"ID":"76b10712-64cc-453d-9a7f-534fc5d8789e","Type":"ContainerDied","Data":"919b53d2a2a4b6390b2fd231b7194124ad55e72aa6a58a6ccb8cffb35c95dbb0"} Sep 16 01:01:05.261919 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:05.261790 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t" event={"ID":"76b10712-64cc-453d-9a7f-534fc5d8789e","Type":"ContainerDied","Data":"0c7eb4b15f284de643ae5fc6fe872d97cd5955fbbb7c4db6e958785cc465d755"} Sep 16 01:01:05.261919 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:05.261807 2567 scope.go:117] "RemoveContainer" containerID="919b53d2a2a4b6390b2fd231b7194124ad55e72aa6a58a6ccb8cffb35c95dbb0" Sep 16 01:01:05.272342 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:05.272320 2567 scope.go:117] "RemoveContainer" containerID="919b53d2a2a4b6390b2fd231b7194124ad55e72aa6a58a6ccb8cffb35c95dbb0" Sep 16 01:01:05.272684 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:01:05.272661 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"919b53d2a2a4b6390b2fd231b7194124ad55e72aa6a58a6ccb8cffb35c95dbb0\": container with ID starting with 919b53d2a2a4b6390b2fd231b7194124ad55e72aa6a58a6ccb8cffb35c95dbb0 not found: ID does not exist" containerID="919b53d2a2a4b6390b2fd231b7194124ad55e72aa6a58a6ccb8cffb35c95dbb0" Sep 16 01:01:05.272747 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:05.272699 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"919b53d2a2a4b6390b2fd231b7194124ad55e72aa6a58a6ccb8cffb35c95dbb0"} err="failed to get container status \"919b53d2a2a4b6390b2fd231b7194124ad55e72aa6a58a6ccb8cffb35c95dbb0\": rpc error: code = NotFound desc = could not find container \"919b53d2a2a4b6390b2fd231b7194124ad55e72aa6a58a6ccb8cffb35c95dbb0\": container with ID starting with 919b53d2a2a4b6390b2fd231b7194124ad55e72aa6a58a6ccb8cffb35c95dbb0 not found: ID does not exist" Sep 16 01:01:05.284085 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:05.284050 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t"] Sep 16 01:01:05.289910 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:05.289865 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2fa86d/rate-test-model-2fa86d-kserve-c94745895-sml9t"] Sep 16 01:01:05.832053 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:05.831998 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:06.120151 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:06.120054 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76b10712-64cc-453d-9a7f-534fc5d8789e" path="/var/lib/kubelet/pods/76b10712-64cc-453d-9a7f-534fc5d8789e/volumes" Sep 16 01:01:10.230710 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:10.230674 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55" Sep 16 01:01:10.405689 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:10.405653 2567 scope.go:117] "RemoveContainer" containerID="af4c7eb83f993ac7e758b9238d33909eb33dc0d2582ecb5c56b6a58ce239d70d" Sep 16 01:01:10.466126 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:10.466091 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:10.576344 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:10.576293 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:10.708328 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:10.708293 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:15.154685 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:15.154644 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:01:19.028953 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:19.028913 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:19.826262 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:19.826222 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:20.520146 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:20.520099 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:21.226672 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:21.226635 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:21.440829 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:21.440789 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:21.539658 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:21.539567 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:22.186787 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:22.186747 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:23.548308 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:23.548271 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:24.251320 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.251289 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:24.538125 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.538007 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws"] Sep 16 01:01:24.538476 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.538449 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b00f081-7f0d-471b-bdd4-85bfe427477f" containerName="istio-proxy" Sep 16 01:01:24.538476 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.538471 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b00f081-7f0d-471b-bdd4-85bfe427477f" containerName="istio-proxy" Sep 16 01:01:24.538704 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.538513 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b10712-64cc-453d-9a7f-534fc5d8789e" containerName="main" Sep 16 01:01:24.538704 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.538521 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b10712-64cc-453d-9a7f-534fc5d8789e" containerName="main" Sep 16 01:01:24.538704 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.538635 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b00f081-7f0d-471b-bdd4-85bfe427477f" containerName="istio-proxy" Sep 16 01:01:24.538704 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.538653 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="76b10712-64cc-453d-9a7f-534fc5d8789e" containerName="main" Sep 16 01:01:24.543521 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.543498 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.546281 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.546252 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0e8a7b-openshift-default-dockercfg-hsw6b\"" Sep 16 01:01:24.553309 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.553269 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws"] Sep 16 01:01:24.566191 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.566150 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:24.633081 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.633006 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-envoy\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.633321 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.633172 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-token\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.633321 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.633206 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-workload-socket\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.633321 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.633251 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a678da49-afd0-4c92-9dd9-07f802b92e3e-istiod-ca-cert\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.633321 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.633274 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v66hb\" (UniqueName: \"kubernetes.io/projected/a678da49-afd0-4c92-9dd9-07f802b92e3e-kube-api-access-v66hb\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.633321 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.633303 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-data\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.633604 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.633344 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-workload-certs\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.633604 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.633448 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-credential-socket\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.633604 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.633570 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-podinfo\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.734580 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.734534 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-workload-certs\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.734806 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.734611 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-credential-socket\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.734806 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.734632 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-podinfo\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.734806 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.734683 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-envoy\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.734806 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.734710 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-token\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.734806 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.734734 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-workload-socket\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.734806 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.734771 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a678da49-afd0-4c92-9dd9-07f802b92e3e-istiod-ca-cert\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.734806 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.734795 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v66hb\" (UniqueName: \"kubernetes.io/projected/a678da49-afd0-4c92-9dd9-07f802b92e3e-kube-api-access-v66hb\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.735242 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.734824 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-data\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.735242 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.735142 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-workload-certs\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.735242 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.735158 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-credential-socket\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.735242 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.735210 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-data\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.735642 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.735618 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a678da49-afd0-4c92-9dd9-07f802b92e3e-istiod-ca-cert\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.735682 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.735645 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-workload-socket\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.737588 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.737556 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-envoy\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.737717 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.737612 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-podinfo\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.743211 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.743186 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-token\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.743351 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.743330 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v66hb\" (UniqueName: \"kubernetes.io/projected/a678da49-afd0-4c92-9dd9-07f802b92e3e-kube-api-access-v66hb\") pod \"e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.856462 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.856421 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:24.901083 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:24.900989 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:25.141015 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:25.140985 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws"] Sep 16 01:01:25.142695 ip-10-0-131-194 kubenswrapper[2567]: W0916 01:01:25.142665 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda678da49_afd0_4c92_9dd9_07f802b92e3e.slice/crio-cbb5367a611e122ea2963b872ad7d1b67a151e7300ae7ee7bd09b23684d01933 WatchSource:0}: Error finding container cbb5367a611e122ea2963b872ad7d1b67a151e7300ae7ee7bd09b23684d01933: Status 404 returned error can't find the container with id cbb5367a611e122ea2963b872ad7d1b67a151e7300ae7ee7bd09b23684d01933 Sep 16 01:01:25.145084 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:25.145049 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:01:25.145194 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:25.145123 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:01:25.145194 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:25.145162 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:01:25.402249 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:25.402028 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" event={"ID":"a678da49-afd0-4c92-9dd9-07f802b92e3e","Type":"ContainerStarted","Data":"7697b8768eb97e8c005cc77f358e609975f6efa0ddb70fb0ae2df25da08a21b6"} Sep 16 01:01:25.402249 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:25.402217 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" event={"ID":"a678da49-afd0-4c92-9dd9-07f802b92e3e","Type":"ContainerStarted","Data":"cbb5367a611e122ea2963b872ad7d1b67a151e7300ae7ee7bd09b23684d01933"} Sep 16 01:01:25.427351 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:25.427287 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" podStartSLOduration=1.427262474 podStartE2EDuration="1.427262474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:01:25.424499496 +0000 UTC m=+1635.786723066" watchObservedRunningTime="2026-09-16 01:01:25.427262474 +0000 UTC m=+1635.789486007" Sep 16 01:01:25.857300 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:25.857260 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:26.632353 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:26.632320 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:26.862512 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:26.862473 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:27.414503 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:27.414471 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:27.415098 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:27.415077 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:27.416227 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:27.416206 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:27.523798 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:27.523760 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:28.269550 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:28.269519 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:29.164327 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:29.164290 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:29.858342 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:29.858307 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:30.645078 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:30.645026 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:30.729796 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:30.729746 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:30.996090 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:30.995925 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:31.123686 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:31.123651 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:31.219344 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:31.219303 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:31.229933 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:31.229892 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr"] Sep 16 01:01:31.230348 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:31.230333 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b00f081-7f0d-471b-bdd4-85bfe427477f" containerName="istio-proxy" Sep 16 01:01:31.230411 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:31.230349 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b00f081-7f0d-471b-bdd4-85bfe427477f" containerName="istio-proxy" Sep 16 01:01:31.230456 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:31.230418 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b00f081-7f0d-471b-bdd4-85bfe427477f" containerName="istio-proxy" Sep 16 01:01:31.233744 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:31.233722 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr" Sep 16 01:01:31.236764 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:31.236741 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0e8a7b\"" Sep 16 01:01:31.236940 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:31.236837 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0e8a7b-dockercfg-kgc7m\"" Sep 16 01:01:31.245281 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:31.245249 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr"] Sep 16 01:01:31.301191 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:31.301153 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1de8ad32-3aca-4ec4-9b85-603c5e895461-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr\" (UID: \"1de8ad32-3aca-4ec4-9b85-603c5e895461\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr" Sep 16 01:01:31.301361 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:31.301325 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh8t9\" (UniqueName: \"kubernetes.io/projected/1de8ad32-3aca-4ec4-9b85-603c5e895461-kube-api-access-nh8t9\") pod \"payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr\" (UID: \"1de8ad32-3aca-4ec4-9b85-603c5e895461\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr" Sep 16 01:01:31.402318 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:31.402280 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh8t9\" (UniqueName: \"kubernetes.io/projected/1de8ad32-3aca-4ec4-9b85-603c5e895461-kube-api-access-nh8t9\") pod \"payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr\" (UID: \"1de8ad32-3aca-4ec4-9b85-603c5e895461\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr" Sep 16 01:01:31.402552 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:31.402346 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1de8ad32-3aca-4ec4-9b85-603c5e895461-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr\" (UID: \"1de8ad32-3aca-4ec4-9b85-603c5e895461\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr" Sep 16 01:01:31.403011 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:31.402991 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1de8ad32-3aca-4ec4-9b85-603c5e895461-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr\" (UID: \"1de8ad32-3aca-4ec4-9b85-603c5e895461\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr" Sep 16 01:01:31.411289 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:31.411261 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh8t9\" (UniqueName: \"kubernetes.io/projected/1de8ad32-3aca-4ec4-9b85-603c5e895461-kube-api-access-nh8t9\") pod \"payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr\" (UID: \"1de8ad32-3aca-4ec4-9b85-603c5e895461\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr" Sep 16 01:01:31.547719 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:31.547675 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr" Sep 16 01:01:31.824740 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:31.824713 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr"] Sep 16 01:01:31.826679 ip-10-0-131-194 kubenswrapper[2567]: W0916 01:01:31.826632 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de8ad32_3aca_4ec4_9b85_603c5e895461.slice/crio-83ea8f2c2d36fcd1375d0bdf266268a2df51d7744c97204fc1092d2f39cc5bab WatchSource:0}: Error finding container 83ea8f2c2d36fcd1375d0bdf266268a2df51d7744c97204fc1092d2f39cc5bab: Status 404 returned error can't find the container with id 83ea8f2c2d36fcd1375d0bdf266268a2df51d7744c97204fc1092d2f39cc5bab Sep 16 01:01:32.449818 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:32.449778 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr" event={"ID":"1de8ad32-3aca-4ec4-9b85-603c5e895461","Type":"ContainerStarted","Data":"c7404f629fe1ba15104e330b8d7fc4a5b701c9567a46f68b38465d466c39d1d1"} Sep 16 01:01:32.450381 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:32.449843 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr" event={"ID":"1de8ad32-3aca-4ec4-9b85-603c5e895461","Type":"ContainerStarted","Data":"83ea8f2c2d36fcd1375d0bdf266268a2df51d7744c97204fc1092d2f39cc5bab"} Sep 16 01:01:32.450381 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:32.450010 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr" Sep 16 01:01:35.237104 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:35.237023 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr" podStartSLOduration=4.23700258 podStartE2EDuration="4.23700258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:01:32.477569381 +0000 UTC m=+1642.839792932" watchObservedRunningTime="2026-09-16 01:01:35.23700258 +0000 UTC m=+1645.599226138" Sep 16 01:01:35.237842 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:35.237805 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:01:42.438889 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:42.438797 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:43.456291 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:43.456242 2567 patch_prober.go:28] interesting pod/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 16 01:01:43.456761 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:43.456346 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" podUID="a678da49-afd0-4c92-9dd9-07f802b92e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 16 01:01:43.458283 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:43.458258 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr" Sep 16 01:01:45.531142 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:45.531020 2567 generic.go:356] "Generic (PLEG): container finished" podID="a678da49-afd0-4c92-9dd9-07f802b92e3e" containerID="7697b8768eb97e8c005cc77f358e609975f6efa0ddb70fb0ae2df25da08a21b6" exitCode=0 Sep 16 01:01:45.531142 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:45.531097 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" event={"ID":"a678da49-afd0-4c92-9dd9-07f802b92e3e","Type":"ContainerDied","Data":"7697b8768eb97e8c005cc77f358e609975f6efa0ddb70fb0ae2df25da08a21b6"} Sep 16 01:01:45.531608 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:45.531528 2567 scope.go:117] "RemoveContainer" containerID="7697b8768eb97e8c005cc77f358e609975f6efa0ddb70fb0ae2df25da08a21b6" Sep 16 01:01:45.532754 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:45.532724 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:01:45.532815 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:45.532802 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:01:45.532869 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:45.532858 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:01:46.539103 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:46.539062 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" event={"ID":"a678da49-afd0-4c92-9dd9-07f802b92e3e","Type":"ContainerStarted","Data":"b7160efc689eaf9e85517becdef22d1cb4a7b730e6e2baf563b6a8a780b4a16c"} Sep 16 01:01:46.856748 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:46.856698 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:46.857119 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:46.857085 2567 patch_prober.go:28] interesting pod/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 16 01:01:46.857397 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:46.857366 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" podUID="a678da49-afd0-4c92-9dd9-07f802b92e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 16 01:01:47.857425 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:47.857385 2567 patch_prober.go:28] interesting pod/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 16 01:01:47.858522 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:47.858476 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" podUID="a678da49-afd0-4c92-9dd9-07f802b92e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 16 01:01:48.014592 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.014551 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:48.078586 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.078549 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:48.387801 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.387768 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:48.675431 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.675388 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf"] Sep 16 01:01:48.679694 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.679665 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.682427 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.682396 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2c31c5-openshift-default-dockercfg-b5j9k\"" Sep 16 01:01:48.690664 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.690626 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf"] Sep 16 01:01:48.698300 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.698267 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:48.785778 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.785739 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-workload-certs\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.786002 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.785788 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b789106-a743-457f-9272-019778117af0-istio-token\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.786002 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.785818 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-workload-socket\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.786002 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.785841 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8txm4\" (UniqueName: \"kubernetes.io/projected/9b789106-a743-457f-9272-019778117af0-kube-api-access-8txm4\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.786002 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.785864 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-istio-data\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.786002 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.785889 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b789106-a743-457f-9272-019778117af0-istio-podinfo\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.786002 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.785918 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b789106-a743-457f-9272-019778117af0-istiod-ca-cert\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.786002 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.785949 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-istio-envoy\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.786002 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.785966 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-credential-socket\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.857916 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.857864 2567 patch_prober.go:28] interesting pod/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 16 01:01:48.858535 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.857958 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" podUID="a678da49-afd0-4c92-9dd9-07f802b92e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 16 01:01:48.887491 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.887443 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-workload-certs\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.887491 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.887495 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b789106-a743-457f-9272-019778117af0-istio-token\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.887780 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.887529 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-workload-socket\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.887780 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.887550 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8txm4\" (UniqueName: \"kubernetes.io/projected/9b789106-a743-457f-9272-019778117af0-kube-api-access-8txm4\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.887780 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.887575 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-istio-data\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.887780 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.887602 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b789106-a743-457f-9272-019778117af0-istio-podinfo\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.888008 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.887952 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-workload-certs\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.888366 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.888033 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b789106-a743-457f-9272-019778117af0-istiod-ca-cert\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.888366 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.888060 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-workload-socket\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.888366 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.888088 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-istio-data\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.888366 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.888139 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-istio-envoy\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.888366 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.888169 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-credential-socket\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.888558 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.888497 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-credential-socket\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.888594 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.888578 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b789106-a743-457f-9272-019778117af0-istiod-ca-cert\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.890474 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.890441 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b789106-a743-457f-9272-019778117af0-istio-podinfo\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.890586 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.890540 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-istio-envoy\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.897163 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.897120 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b789106-a743-457f-9272-019778117af0-istio-token\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.897368 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.897341 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8txm4\" (UniqueName: \"kubernetes.io/projected/9b789106-a743-457f-9272-019778117af0-kube-api-access-8txm4\") pod \"e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:48.995722 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:48.995624 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:49.135307 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:49.135270 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:49.307473 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:49.307440 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf"] Sep 16 01:01:49.309118 ip-10-0-131-194 kubenswrapper[2567]: W0916 01:01:49.309085 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b789106_a743_457f_9272_019778117af0.slice/crio-cb59ed01df361a9848bc82a0fdc3aced7ed02c34fbe5fdf39572f506fab3263b WatchSource:0}: Error finding container cb59ed01df361a9848bc82a0fdc3aced7ed02c34fbe5fdf39572f506fab3263b: Status 404 returned error can't find the container with id cb59ed01df361a9848bc82a0fdc3aced7ed02c34fbe5fdf39572f506fab3263b Sep 16 01:01:49.311733 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:49.311695 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:01:49.311840 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:49.311776 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:01:49.311840 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:49.311806 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:01:49.570602 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:49.570417 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" event={"ID":"9b789106-a743-457f-9272-019778117af0","Type":"ContainerStarted","Data":"9e9e9b1edb7bd925ced8bd4f298a1e04e4278598138f7361f93c68e13b5d9a85"} Sep 16 01:01:49.570602 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:49.570498 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" event={"ID":"9b789106-a743-457f-9272-019778117af0","Type":"ContainerStarted","Data":"cb59ed01df361a9848bc82a0fdc3aced7ed02c34fbe5fdf39572f506fab3263b"} Sep 16 01:01:49.593889 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:49.593834 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" podStartSLOduration=1.593807593 podStartE2EDuration="1.593807593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:01:49.592823153 +0000 UTC m=+1659.955046710" watchObservedRunningTime="2026-09-16 01:01:49.593807593 +0000 UTC m=+1659.956031123" Sep 16 01:01:49.857444 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:49.857326 2567 patch_prober.go:28] interesting pod/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 16 01:01:49.857444 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:49.857405 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" podUID="a678da49-afd0-4c92-9dd9-07f802b92e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 16 01:01:49.996533 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:49.996484 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:50.857387 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:50.857334 2567 patch_prober.go:28] interesting pod/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 16 01:01:50.857627 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:50.857416 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" podUID="a678da49-afd0-4c92-9dd9-07f802b92e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 16 01:01:51.001751 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:51.001713 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:51.533634 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:51.533591 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:51.585380 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:51.585339 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:51.586640 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:51.586613 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:01:51.615244 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:51.615203 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:51.796241 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:51.796143 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:51.951259 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:51.951226 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:51.952027 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:51.952001 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:51.953962 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:51.953934 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:01:52.430788 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:52.430751 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:52.540886 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:52.540844 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:53.541366 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:53.541069 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:54.327416 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:54.327386 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:55.014544 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:55.014510 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:55.938523 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:55.938488 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:56.626753 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:56.626713 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:56.727161 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:56.727123 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:57.037695 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:57.037584 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:57.135833 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:57.135791 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:57.202704 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:57.202668 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:57.774857 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:57.774819 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:01:58.425937 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:01:58.425906 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:01.704741 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.704705 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp"] Sep 16 01:02:01.708690 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.708668 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:01.713121 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.713092 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-715834\"/\"ipp-route-715834-kserve-self-signed-certs\"" Sep 16 01:02:01.713311 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.713093 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-715834\"/\"openshift-service-ca.crt\"" Sep 16 01:02:01.713311 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.713135 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-715834\"/\"default-dockercfg-2zp24\"" Sep 16 01:02:01.713311 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.713121 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-715834\"/\"kube-root-ca.crt\"" Sep 16 01:02:01.720148 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.720114 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp"] Sep 16 01:02:01.761991 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.761962 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:01.821689 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.821651 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfcd3554-65e7-45c1-94f5-df99995fead2-tls-certs\") pod \"ipp-route-715834-kserve-6b4b598d59-6tbhp\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:01.821882 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.821731 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-home\") pod \"ipp-route-715834-kserve-6b4b598d59-6tbhp\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:01.821882 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.821796 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-dshm\") pod \"ipp-route-715834-kserve-6b4b598d59-6tbhp\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:01.821882 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.821825 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-model-cache\") pod \"ipp-route-715834-kserve-6b4b598d59-6tbhp\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:01.821882 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.821849 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lq8n\" (UniqueName: \"kubernetes.io/projected/cfcd3554-65e7-45c1-94f5-df99995fead2-kube-api-access-5lq8n\") pod \"ipp-route-715834-kserve-6b4b598d59-6tbhp\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:01.822133 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.821913 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-tmp-dir\") pod \"ipp-route-715834-kserve-6b4b598d59-6tbhp\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:01.923415 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.923370 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfcd3554-65e7-45c1-94f5-df99995fead2-tls-certs\") pod \"ipp-route-715834-kserve-6b4b598d59-6tbhp\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:01.923628 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.923431 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-home\") pod \"ipp-route-715834-kserve-6b4b598d59-6tbhp\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:01.923628 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.923448 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-dshm\") pod \"ipp-route-715834-kserve-6b4b598d59-6tbhp\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:01.923628 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.923475 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-model-cache\") pod \"ipp-route-715834-kserve-6b4b598d59-6tbhp\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:01.923628 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.923495 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lq8n\" (UniqueName: \"kubernetes.io/projected/cfcd3554-65e7-45c1-94f5-df99995fead2-kube-api-access-5lq8n\") pod \"ipp-route-715834-kserve-6b4b598d59-6tbhp\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:01.923628 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.923529 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-tmp-dir\") pod \"ipp-route-715834-kserve-6b4b598d59-6tbhp\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:01.923897 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.923862 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-home\") pod \"ipp-route-715834-kserve-6b4b598d59-6tbhp\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:01.923945 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.923933 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-tmp-dir\") pod \"ipp-route-715834-kserve-6b4b598d59-6tbhp\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:01.923980 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.923947 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-model-cache\") pod \"ipp-route-715834-kserve-6b4b598d59-6tbhp\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:01.926134 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.926109 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-dshm\") pod \"ipp-route-715834-kserve-6b4b598d59-6tbhp\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:01.926411 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.926391 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfcd3554-65e7-45c1-94f5-df99995fead2-tls-certs\") pod \"ipp-route-715834-kserve-6b4b598d59-6tbhp\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:01.931951 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:01.931928 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lq8n\" (UniqueName: \"kubernetes.io/projected/cfcd3554-65e7-45c1-94f5-df99995fead2-kube-api-access-5lq8n\") pod \"ipp-route-715834-kserve-6b4b598d59-6tbhp\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:02.020597 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:02.020492 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:02.302586 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:02.302553 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp"] Sep 16 01:02:02.303946 ip-10-0-131-194 kubenswrapper[2567]: W0916 01:02:02.303916 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfcd3554_65e7_45c1_94f5_df99995fead2.slice/crio-87a824332c98dd5bfed6e1823af3a7f8c173a63869142fe58ab6174db05723e5 WatchSource:0}: Error finding container 87a824332c98dd5bfed6e1823af3a7f8c173a63869142fe58ab6174db05723e5: Status 404 returned error can't find the container with id 87a824332c98dd5bfed6e1823af3a7f8c173a63869142fe58ab6174db05723e5 Sep 16 01:02:02.644483 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:02.644388 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:02:02.654225 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:02.654178 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" event={"ID":"cfcd3554-65e7-45c1-94f5-df99995fead2","Type":"ContainerStarted","Data":"4339494f3d697fd10f3aaa549d14cf5e1c74979452ff25a4836a3ec0a1e45378"} Sep 16 01:02:02.654380 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:02.654237 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" event={"ID":"cfcd3554-65e7-45c1-94f5-df99995fead2","Type":"ContainerStarted","Data":"87a824332c98dd5bfed6e1823af3a7f8c173a63869142fe58ab6174db05723e5"} Sep 16 01:02:02.672411 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:02.672345 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" podStartSLOduration=1.672327685 podStartE2EDuration="1.672327685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:02:02.670636292 +0000 UTC m=+1673.032859824" watchObservedRunningTime="2026-09-16 01:02:02.672327685 +0000 UTC m=+1673.034551218" Sep 16 01:02:12.021000 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:12.020954 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:12.021000 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:12.021012 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:12.034537 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:12.034506 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:12.728144 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:12.728101 2567 generic.go:356] "Generic (PLEG): container finished" podID="9b789106-a743-457f-9272-019778117af0" containerID="9e9e9b1edb7bd925ced8bd4f298a1e04e4278598138f7361f93c68e13b5d9a85" exitCode=0 Sep 16 01:02:12.728367 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:12.728176 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" event={"ID":"9b789106-a743-457f-9272-019778117af0","Type":"ContainerDied","Data":"9e9e9b1edb7bd925ced8bd4f298a1e04e4278598138f7361f93c68e13b5d9a85"} Sep 16 01:02:12.728687 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:12.728661 2567 scope.go:117] "RemoveContainer" containerID="9e9e9b1edb7bd925ced8bd4f298a1e04e4278598138f7361f93c68e13b5d9a85" Sep 16 01:02:12.730049 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:12.730007 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:02:12.730129 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:12.730102 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:02:12.730167 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:12.730138 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:02:12.740742 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:12.740705 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:02:13.225478 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:13.225444 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:13.736645 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:13.736607 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" event={"ID":"9b789106-a743-457f-9272-019778117af0","Type":"ContainerStarted","Data":"b8fb534c5b0d318e37c652a6d9481c4f032bc0ad6ea5f9e4bd4e4fc4b051e868"} Sep 16 01:02:13.996847 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:13.996736 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:02:13.997082 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:13.997020 2567 patch_prober.go:28] interesting pod/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Sep 16 01:02:13.997187 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:13.997135 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" podUID="9b789106-a743-457f-9272-019778117af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Sep 16 01:02:14.997124 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:14.997089 2567 patch_prober.go:28] interesting pod/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Sep 16 01:02:14.997653 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:14.997606 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" podUID="9b789106-a743-457f-9272-019778117af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Sep 16 01:02:15.996735 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:15.996696 2567 patch_prober.go:28] interesting pod/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Sep 16 01:02:15.996957 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:15.996771 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" podUID="9b789106-a743-457f-9272-019778117af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Sep 16 01:02:16.996315 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:16.996272 2567 patch_prober.go:28] interesting pod/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Sep 16 01:02:16.996733 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:16.996338 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" podUID="9b789106-a743-457f-9272-019778117af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Sep 16 01:02:17.500413 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:17.500369 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:17.996962 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:17.996914 2567 patch_prober.go:28] interesting pod/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Sep 16 01:02:17.997447 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:17.996983 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" podUID="9b789106-a743-457f-9272-019778117af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Sep 16 01:02:19.054859 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:19.054804 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:19.083717 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:19.083677 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:02:19.084226 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:19.084196 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:02:19.085286 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:19.085264 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:02:19.715533 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:19.715494 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:19.740381 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:19.740342 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:02:20.538233 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:20.538202 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:21.154901 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:21.154862 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:21.650099 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:21.650067 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:21.922468 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:21.922380 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:22.135725 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.135668 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:22.483022 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.482968 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n"] Sep 16 01:02:22.489000 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.488968 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.491951 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.491920 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-761551-openshift-default-dockercfg-whww8\"" Sep 16 01:02:22.495594 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.495563 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n"] Sep 16 01:02:22.501802 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.501770 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:22.549326 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.549279 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-workload-certs\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.549534 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.549340 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-podinfo\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.549534 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.549421 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-envoy\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.549534 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.549475 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-credential-socket\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.549660 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.549534 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-token\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.549660 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.549572 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5kh2\" (UniqueName: \"kubernetes.io/projected/124e2e72-5d99-4745-92a0-a6624da3ea07-kube-api-access-t5kh2\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.549660 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.549595 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-data\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.549660 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.549637 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/124e2e72-5d99-4745-92a0-a6624da3ea07-istiod-ca-cert\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.549811 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.549674 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-workload-socket\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.650968 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.650920 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-workload-certs\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.650968 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.650972 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-podinfo\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.651574 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.651031 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-envoy\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.651574 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.651091 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-credential-socket\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.651574 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.651119 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-token\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.651574 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.651161 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5kh2\" (UniqueName: \"kubernetes.io/projected/124e2e72-5d99-4745-92a0-a6624da3ea07-kube-api-access-t5kh2\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.651574 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.651191 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-data\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.651574 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.651216 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/124e2e72-5d99-4745-92a0-a6624da3ea07-istiod-ca-cert\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.651574 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.651248 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-workload-socket\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.651574 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.651438 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-workload-certs\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.652209 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.651606 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-data\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.652209 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.651645 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-workload-socket\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.652209 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.651797 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-credential-socket\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.652209 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.652104 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/124e2e72-5d99-4745-92a0-a6624da3ea07-istiod-ca-cert\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.653716 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.653693 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-envoy\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.653847 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.653830 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-podinfo\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.661090 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.661007 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5kh2\" (UniqueName: \"kubernetes.io/projected/124e2e72-5d99-4745-92a0-a6624da3ea07-kube-api-access-t5kh2\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.661090 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.661071 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-token\") pod \"e2e-api-b-761551-openshift-default-54d7857944-bg55n\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.712842 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.712810 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:22.802054 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.802004 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:22.900866 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:22.900834 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:23.092462 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:23.092433 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n"] Sep 16 01:02:23.094844 ip-10-0-131-194 kubenswrapper[2567]: W0916 01:02:23.094810 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124e2e72_5d99_4745_92a0_a6624da3ea07.slice/crio-00790d69ac0c0d92b6bf8d207022f3ee639f0c6569e5b84b3a154f66b1131a75 WatchSource:0}: Error finding container 00790d69ac0c0d92b6bf8d207022f3ee639f0c6569e5b84b3a154f66b1131a75: Status 404 returned error can't find the container with id 00790d69ac0c0d92b6bf8d207022f3ee639f0c6569e5b84b3a154f66b1131a75 Sep 16 01:02:23.097312 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:23.097276 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:02:23.097414 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:23.097346 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:02:23.097414 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:23.097378 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:02:23.575594 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:23.575560 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:23.814458 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:23.814414 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" event={"ID":"124e2e72-5d99-4745-92a0-a6624da3ea07","Type":"ContainerStarted","Data":"6c471907d5e14eddf605efad0d7438bcaece6be2937da7bd3e5394e5af82d6ba"} Sep 16 01:02:23.814898 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:23.814485 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" event={"ID":"124e2e72-5d99-4745-92a0-a6624da3ea07","Type":"ContainerStarted","Data":"00790d69ac0c0d92b6bf8d207022f3ee639f0c6569e5b84b3a154f66b1131a75"} Sep 16 01:02:23.837174 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:23.837030 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" podStartSLOduration=1.837011558 podStartE2EDuration="1.837011558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:02:23.83512983 +0000 UTC m=+1694.197353383" watchObservedRunningTime="2026-09-16 01:02:23.837011558 +0000 UTC m=+1694.199235090" Sep 16 01:02:24.803176 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:24.803139 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:24.809461 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:24.809426 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:24.820539 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:24.820507 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:24.821577 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:24.821556 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:25.527583 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:25.527520 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:25.873868 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:25.873803 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:25.972004 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:25.971958 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:26.266568 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:26.266481 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:27.055274 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:27.055234 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:27.823764 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:27.823719 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:28.611265 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:28.611226 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:29.321670 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:29.321636 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:29.974563 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:29.974524 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:30.744271 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:30.744240 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:30.845351 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:30.845283 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:31.204602 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:31.204550 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:31.289780 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:31.289696 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:31.367889 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:31.367831 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:39.242517 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:39.242466 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:02:44.341159 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:44.341110 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:46.797306 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:46.797274 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55_21cb4d08-16e2-4d5a-9f9e-59c394282a26/payload-processing/0.log" Sep 16 01:02:47.598105 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:47.598071 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:47.933741 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:47.933641 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994"] Sep 16 01:02:47.936746 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:47.936718 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:47.939869 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:47.939842 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-eba3d3-openshift-default-dockercfg-7h7pf\"" Sep 16 01:02:47.953418 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:47.953387 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994"] Sep 16 01:02:47.962834 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:47.962780 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:48.031956 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.031909 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-envoy\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.032213 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.032004 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n97d9\" (UniqueName: \"kubernetes.io/projected/1aba147f-9352-4b2b-9e23-7eac729a8fd8-kube-api-access-n97d9\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.032213 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.032063 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-podinfo\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.032213 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.032091 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.032213 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.032179 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-workload-socket\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.032462 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.032250 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-credential-socket\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.032462 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.032277 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-workload-certs\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.032462 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.032298 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-data\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.032462 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.032395 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-token\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.133015 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.132976 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n97d9\" (UniqueName: \"kubernetes.io/projected/1aba147f-9352-4b2b-9e23-7eac729a8fd8-kube-api-access-n97d9\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.133015 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.133022 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-podinfo\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.133297 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.133102 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.133297 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.133183 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-workload-socket\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.133384 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.133337 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-credential-socket\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.133442 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.133383 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-workload-certs\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.133442 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.133405 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-data\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.133544 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.133502 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-token\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.133544 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.133538 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-envoy\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.133654 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.133635 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-workload-socket\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.133772 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.133738 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-workload-certs\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.133985 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.133936 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-data\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.134209 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.134102 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-credential-socket\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.134274 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.134221 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.135858 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.135833 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-envoy\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.136099 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.136075 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-podinfo\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.145958 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.145912 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-token\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.146176 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.146157 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n97d9\" (UniqueName: \"kubernetes.io/projected/1aba147f-9352-4b2b-9e23-7eac729a8fd8-kube-api-access-n97d9\") pod \"e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.186668 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.186571 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:48.252353 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.252317 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:48.537669 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.537634 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994"] Sep 16 01:02:48.539656 ip-10-0-131-194 kubenswrapper[2567]: W0916 01:02:48.539625 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aba147f_9352_4b2b_9e23_7eac729a8fd8.slice/crio-8ca65719ea45fc8d12de64aeb94eb90207ae92b7f117cec94dacb2c3fb1a06ef WatchSource:0}: Error finding container 8ca65719ea45fc8d12de64aeb94eb90207ae92b7f117cec94dacb2c3fb1a06ef: Status 404 returned error can't find the container with id 8ca65719ea45fc8d12de64aeb94eb90207ae92b7f117cec94dacb2c3fb1a06ef Sep 16 01:02:48.542758 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.542714 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:02:48.542855 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.542818 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:02:48.542895 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.542859 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:02:48.996419 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.996381 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" event={"ID":"1aba147f-9352-4b2b-9e23-7eac729a8fd8","Type":"ContainerStarted","Data":"211ef5dcb6870bef843fb4d5c48ba9660fdc2a1b67793aacd55f5edb62034289"} Sep 16 01:02:48.996826 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:48.996443 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" event={"ID":"1aba147f-9352-4b2b-9e23-7eac729a8fd8","Type":"ContainerStarted","Data":"8ca65719ea45fc8d12de64aeb94eb90207ae92b7f117cec94dacb2c3fb1a06ef"} Sep 16 01:02:49.019723 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:49.019664 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" podStartSLOduration=2.019646406 podStartE2EDuration="2.019646406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:02:49.016826729 +0000 UTC m=+1719.379050282" watchObservedRunningTime="2026-09-16 01:02:49.019646406 +0000 UTC m=+1719.381870100" Sep 16 01:02:49.253254 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:49.253164 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:49.259565 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:49.259537 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:50.004825 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:50.004788 2567 generic.go:356] "Generic (PLEG): container finished" podID="124e2e72-5d99-4745-92a0-a6624da3ea07" containerID="6c471907d5e14eddf605efad0d7438bcaece6be2937da7bd3e5394e5af82d6ba" exitCode=0 Sep 16 01:02:50.005300 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:50.004857 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" event={"ID":"124e2e72-5d99-4745-92a0-a6624da3ea07","Type":"ContainerDied","Data":"6c471907d5e14eddf605efad0d7438bcaece6be2937da7bd3e5394e5af82d6ba"} Sep 16 01:02:50.005346 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:50.005305 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:50.005433 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:50.005418 2567 scope.go:117] "RemoveContainer" containerID="6c471907d5e14eddf605efad0d7438bcaece6be2937da7bd3e5394e5af82d6ba" Sep 16 01:02:50.006601 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:50.006571 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:02:50.006726 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:50.006699 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:02:50.006780 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:50.006768 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:02:50.006817 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:50.006810 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:02:50.336507 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:50.336472 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:51.015094 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:51.015063 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:51.015608 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:51.015422 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" event={"ID":"124e2e72-5d99-4745-92a0-a6624da3ea07","Type":"ContainerStarted","Data":"50de2a8d95d58a99b28e8ab6e2c398c7b933e38c9926cb06a29f741493f36e73"} Sep 16 01:02:51.113553 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:51.113527 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:51.803225 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:51.803182 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:51.804601 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:51.804573 2567 patch_prober.go:28] interesting pod/e2e-api-b-761551-openshift-default-54d7857944-bg55n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 16 01:02:51.804749 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:51.804646 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" podUID="124e2e72-5d99-4745-92a0-a6624da3ea07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 16 01:02:52.000390 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:52.000353 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:52.764767 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:52.764727 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:52.803442 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:52.803407 2567 patch_prober.go:28] interesting pod/e2e-api-b-761551-openshift-default-54d7857944-bg55n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 16 01:02:52.803657 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:52.803481 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" podUID="124e2e72-5d99-4745-92a0-a6624da3ea07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 16 01:02:53.519623 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:53.519551 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:53.802702 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:53.802655 2567 patch_prober.go:28] interesting pod/e2e-api-b-761551-openshift-default-54d7857944-bg55n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 16 01:02:53.803228 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:53.802721 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" podUID="124e2e72-5d99-4745-92a0-a6624da3ea07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 16 01:02:54.471280 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:54.471250 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:54.803139 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:54.803090 2567 patch_prober.go:28] interesting pod/e2e-api-b-761551-openshift-default-54d7857944-bg55n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 16 01:02:54.803655 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:54.803166 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" podUID="124e2e72-5d99-4745-92a0-a6624da3ea07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 16 01:02:54.820613 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:54.820571 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:55.322319 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:55.322288 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:55.802825 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:55.802790 2567 patch_prober.go:28] interesting pod/e2e-api-b-761551-openshift-default-54d7857944-bg55n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 16 01:02:55.803013 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:55.802853 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" podUID="124e2e72-5d99-4745-92a0-a6624da3ea07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 16 01:02:55.919359 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:55.919324 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:56.024844 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:56.024810 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:56.728269 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:56.728201 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:56.808494 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:56.808459 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:56.809692 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:56.809670 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:02:57.457609 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:57.457571 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:57.559750 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:57.559714 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:57.769834 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:57.769745 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:57.936875 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:57.936840 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:58.034895 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:58.034803 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:58.078494 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:58.078456 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l"] Sep 16 01:02:58.081603 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:58.081580 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l" Sep 16 01:02:58.084244 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:58.084220 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-eba3d3-dockercfg-hx5mc\"" Sep 16 01:02:58.084411 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:58.084227 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-eba3d3\"" Sep 16 01:02:58.092825 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:58.092793 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l"] Sep 16 01:02:58.163235 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:58.163187 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16067f7e-16b1-42e4-8bea-54a88f0ffbd9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l\" (UID: \"16067f7e-16b1-42e4-8bea-54a88f0ffbd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l" Sep 16 01:02:58.163559 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:58.163538 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f58fz\" (UniqueName: \"kubernetes.io/projected/16067f7e-16b1-42e4-8bea-54a88f0ffbd9-kube-api-access-f58fz\") pod \"payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l\" (UID: \"16067f7e-16b1-42e4-8bea-54a88f0ffbd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l" Sep 16 01:02:58.265449 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:58.265411 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f58fz\" (UniqueName: \"kubernetes.io/projected/16067f7e-16b1-42e4-8bea-54a88f0ffbd9-kube-api-access-f58fz\") pod \"payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l\" (UID: \"16067f7e-16b1-42e4-8bea-54a88f0ffbd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l" Sep 16 01:02:58.265677 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:58.265658 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16067f7e-16b1-42e4-8bea-54a88f0ffbd9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l\" (UID: \"16067f7e-16b1-42e4-8bea-54a88f0ffbd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l" Sep 16 01:02:58.266396 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:58.266378 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16067f7e-16b1-42e4-8bea-54a88f0ffbd9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l\" (UID: \"16067f7e-16b1-42e4-8bea-54a88f0ffbd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l" Sep 16 01:02:58.273811 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:58.273781 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f58fz\" (UniqueName: \"kubernetes.io/projected/16067f7e-16b1-42e4-8bea-54a88f0ffbd9-kube-api-access-f58fz\") pod \"payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l\" (UID: \"16067f7e-16b1-42e4-8bea-54a88f0ffbd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l" Sep 16 01:02:58.330409 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:58.330374 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:02:58.393729 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:58.393688 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l" Sep 16 01:02:58.678558 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:58.678533 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l"] Sep 16 01:02:58.680197 ip-10-0-131-194 kubenswrapper[2567]: W0916 01:02:58.680167 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16067f7e_16b1_42e4_8bea_54a88f0ffbd9.slice/crio-c3dd60bdf85f0ad4b1495ea48a4212be8f71a60b6bf459b2e46a7eca8be9963d WatchSource:0}: Error finding container c3dd60bdf85f0ad4b1495ea48a4212be8f71a60b6bf459b2e46a7eca8be9963d: Status 404 returned error can't find the container with id c3dd60bdf85f0ad4b1495ea48a4212be8f71a60b6bf459b2e46a7eca8be9963d Sep 16 01:02:59.073424 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:59.073389 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l" event={"ID":"16067f7e-16b1-42e4-8bea-54a88f0ffbd9","Type":"ContainerStarted","Data":"8d01900c823a4e5cace9ae4451eb270505fb5029a55c525272ad104e6b192baf"} Sep 16 01:02:59.073659 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:59.073449 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l" event={"ID":"16067f7e-16b1-42e4-8bea-54a88f0ffbd9","Type":"ContainerStarted","Data":"c3dd60bdf85f0ad4b1495ea48a4212be8f71a60b6bf459b2e46a7eca8be9963d"} Sep 16 01:02:59.073659 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:59.073581 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l" Sep 16 01:02:59.093375 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:59.093318 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l" podStartSLOduration=1.09330011 podStartE2EDuration="1.09330011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:02:59.09167764 +0000 UTC m=+1729.453901173" watchObservedRunningTime="2026-09-16 01:02:59.09330011 +0000 UTC m=+1729.455523639" Sep 16 01:02:59.323998 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:02:59.323905 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:00.006422 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:00.006388 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:00.822517 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:00.822478 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:01.625686 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:01.625639 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:02.440659 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:02.440627 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:03.096426 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:03.096386 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:03.747549 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:03.747518 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:04.537752 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:04.537719 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:05.206681 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:05.206645 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:05.977377 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:05.977340 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:07.458459 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:07.458421 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:07.536740 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:07.536705 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:03:08.247481 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:08.247445 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:09.040769 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:09.040736 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:09.823687 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:09.823650 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:10.090572 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:10.090471 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l" Sep 16 01:03:10.471260 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:10.471157 2567 scope.go:117] "RemoveContainer" containerID="b4eb3a1748892480ef22db18957f0d3c9d09701f282638b700b1f37d44c08e16" Sep 16 01:03:10.494406 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:10.494362 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:10.868794 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:10.868750 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:11.348287 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:11.348247 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:12.231463 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:12.231429 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:12.907468 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:12.907428 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:13.719939 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:13.719886 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:14.424590 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:14.424528 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:15.362071 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:15.362010 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:16.028917 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:16.028857 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:16.839975 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:16.839941 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:17.574865 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:17.574823 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:18.225772 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:18.225737 2567 generic.go:356] "Generic (PLEG): container finished" podID="1aba147f-9352-4b2b-9e23-7eac729a8fd8" containerID="211ef5dcb6870bef843fb4d5c48ba9660fdc2a1b67793aacd55f5edb62034289" exitCode=0 Sep 16 01:03:18.226012 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:18.225796 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" event={"ID":"1aba147f-9352-4b2b-9e23-7eac729a8fd8","Type":"ContainerDied","Data":"211ef5dcb6870bef843fb4d5c48ba9660fdc2a1b67793aacd55f5edb62034289"} Sep 16 01:03:18.226283 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:18.226267 2567 scope.go:117] "RemoveContainer" containerID="211ef5dcb6870bef843fb4d5c48ba9660fdc2a1b67793aacd55f5edb62034289" Sep 16 01:03:18.227620 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:18.227589 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:03:18.227678 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:18.227665 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:03:18.227713 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:18.227698 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 16 01:03:18.312227 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:18.312192 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:19.233744 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:19.233707 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" event={"ID":"1aba147f-9352-4b2b-9e23-7eac729a8fd8","Type":"ContainerStarted","Data":"51f4b9b8c8294bf14e7851abb7c9ca7868e1e8a9fe924f311c3791ac7cc8f06a"} Sep 16 01:03:19.253486 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:19.253444 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:03:19.254911 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:19.254875 2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Sep 16 01:03:19.255244 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:19.255207 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" podUID="1aba147f-9352-4b2b-9e23-7eac729a8fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Sep 16 01:03:19.300052 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:19.300000 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:19.376534 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:19.376500 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:19.438479 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:19.438443 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:19.507983 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:19.507892 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:19.569212 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:19.569171 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:19.883174 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:19.883146 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:20.005465 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:20.005421 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:03:20.036827 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:20.036782 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:20.253416 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:20.253305 2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Sep 16 01:03:20.254068 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:20.254004 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" podUID="1aba147f-9352-4b2b-9e23-7eac729a8fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Sep 16 01:03:20.712473 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:20.712428 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:21.253267 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:21.253219 2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Sep 16 01:03:21.253500 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:21.253294 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" podUID="1aba147f-9352-4b2b-9e23-7eac729a8fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Sep 16 01:03:21.413239 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:21.413204 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:22.115737 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:22.115705 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:22.255310 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:22.255258 2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Sep 16 01:03:22.256734 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:22.256014 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" podUID="1aba147f-9352-4b2b-9e23-7eac729a8fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Sep 16 01:03:22.920141 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:22.920103 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:23.253231 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:23.253113 2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Sep 16 01:03:23.253559 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:23.253522 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" podUID="1aba147f-9352-4b2b-9e23-7eac729a8fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Sep 16 01:03:23.749622 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:23.749548 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:24.253670 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:24.253627 2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Sep 16 01:03:24.254026 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:24.253995 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" podUID="1aba147f-9352-4b2b-9e23-7eac729a8fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Sep 16 01:03:24.437599 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:24.437530 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:25.021532 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:25.021491 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:25.055070 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:25.055016 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l"] Sep 16 01:03:25.055384 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:25.055335 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l" podUID="16067f7e-16b1-42e4-8bea-54a88f0ffbd9" containerName="payload-processing" containerID="cri-o://8d01900c823a4e5cace9ae4451eb270505fb5029a55c525272ad104e6b192baf" gracePeriod=30 Sep 16 01:03:25.091125 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:25.091084 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:25.191953 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:25.191888 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:25.258262 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:25.258222 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:25.259805 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:25.259780 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:03:25.261087 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:25.261065 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:03:25.293740 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:25.293651 2567 generic.go:356] "Generic (PLEG): container finished" podID="16067f7e-16b1-42e4-8bea-54a88f0ffbd9" containerID="8d01900c823a4e5cace9ae4451eb270505fb5029a55c525272ad104e6b192baf" exitCode=0 Sep 16 01:03:25.293740 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:25.293716 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l" event={"ID":"16067f7e-16b1-42e4-8bea-54a88f0ffbd9","Type":"ContainerDied","Data":"8d01900c823a4e5cace9ae4451eb270505fb5029a55c525272ad104e6b192baf"} Sep 16 01:03:25.327945 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:25.327917 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l" Sep 16 01:03:25.470799 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:25.470752 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16067f7e-16b1-42e4-8bea-54a88f0ffbd9-plugins-config-volume\") pod \"16067f7e-16b1-42e4-8bea-54a88f0ffbd9\" (UID: \"16067f7e-16b1-42e4-8bea-54a88f0ffbd9\") " Sep 16 01:03:25.470992 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:25.470859 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f58fz\" (UniqueName: \"kubernetes.io/projected/16067f7e-16b1-42e4-8bea-54a88f0ffbd9-kube-api-access-f58fz\") pod \"16067f7e-16b1-42e4-8bea-54a88f0ffbd9\" (UID: \"16067f7e-16b1-42e4-8bea-54a88f0ffbd9\") " Sep 16 01:03:25.471184 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:25.471162 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16067f7e-16b1-42e4-8bea-54a88f0ffbd9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "16067f7e-16b1-42e4-8bea-54a88f0ffbd9" (UID: "16067f7e-16b1-42e4-8bea-54a88f0ffbd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:03:25.473357 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:25.473332 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16067f7e-16b1-42e4-8bea-54a88f0ffbd9-kube-api-access-f58fz" (OuterVolumeSpecName: "kube-api-access-f58fz") pod "16067f7e-16b1-42e4-8bea-54a88f0ffbd9" (UID: "16067f7e-16b1-42e4-8bea-54a88f0ffbd9"). InnerVolumeSpecName "kube-api-access-f58fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:03:25.572214 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:25.572170 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16067f7e-16b1-42e4-8bea-54a88f0ffbd9-plugins-config-volume\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:25.572214 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:25.572208 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f58fz\" (UniqueName: \"kubernetes.io/projected/16067f7e-16b1-42e4-8bea-54a88f0ffbd9-kube-api-access-f58fz\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:25.650415 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:25.650377 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:25.749201 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:25.749160 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:26.303397 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:26.303369 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l" Sep 16 01:03:26.303397 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:26.303378 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l" event={"ID":"16067f7e-16b1-42e4-8bea-54a88f0ffbd9","Type":"ContainerDied","Data":"c3dd60bdf85f0ad4b1495ea48a4212be8f71a60b6bf459b2e46a7eca8be9963d"} Sep 16 01:03:26.303920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:26.303465 2567 scope.go:117] "RemoveContainer" containerID="8d01900c823a4e5cace9ae4451eb270505fb5029a55c525272ad104e6b192baf" Sep 16 01:03:26.321810 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:26.321772 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l"] Sep 16 01:03:26.326012 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:26.325985 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eba3d3-6b8cbd54b7-vpb7l"] Sep 16 01:03:26.541598 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:26.541487 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:26.966195 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:26.966142 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:27.029206 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:27.029125 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:27.225848 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:27.225675 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf"] Sep 16 01:03:27.226063 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:27.225962 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" podUID="9b789106-a743-457f-9272-019778117af0" containerName="istio-proxy" containerID="cri-o://b8fb534c5b0d318e37c652a6d9481c4f032bc0ad6ea5f9e4bd4e4fc4b051e868" gracePeriod=30 Sep 16 01:03:27.238074 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:27.238015 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:27.400261 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:27.400217 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:28.119096 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:28.119063 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16067f7e-16b1-42e4-8bea-54a88f0ffbd9" path="/var/lib/kubelet/pods/16067f7e-16b1-42e4-8bea-54a88f0ffbd9/volumes" Sep 16 01:03:28.451058 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:28.450948 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:29.328613 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:29.328573 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:30.018367 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:30.018327 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:30.236176 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:30.236127 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:03:30.834105 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:30.834079 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:31.571242 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:31.571203 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:32.105676 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.105637 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:32.212565 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.212526 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:32.305935 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.305901 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:32.369186 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.369024 2567 generic.go:356] "Generic (PLEG): container finished" podID="9b789106-a743-457f-9272-019778117af0" containerID="b8fb534c5b0d318e37c652a6d9481c4f032bc0ad6ea5f9e4bd4e4fc4b051e868" exitCode=0 Sep 16 01:03:32.369417 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.369177 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" event={"ID":"9b789106-a743-457f-9272-019778117af0","Type":"ContainerDied","Data":"b8fb534c5b0d318e37c652a6d9481c4f032bc0ad6ea5f9e4bd4e4fc4b051e868"} Sep 16 01:03:32.369417 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.369255 2567 scope.go:117] "RemoveContainer" containerID="9e9e9b1edb7bd925ced8bd4f298a1e04e4278598138f7361f93c68e13b5d9a85" Sep 16 01:03:32.392950 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.392732 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994"] Sep 16 01:03:32.393546 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.393403 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" podUID="1aba147f-9352-4b2b-9e23-7eac729a8fd8" containerName="istio-proxy" containerID="cri-o://51f4b9b8c8294bf14e7851abb7c9ca7868e1e8a9fe924f311c3791ac7cc8f06a" gracePeriod=30 Sep 16 01:03:32.399841 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.399812 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:32.550319 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.550289 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:03:32.654971 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.654870 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-istio-envoy\") pod \"9b789106-a743-457f-9272-019778117af0\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " Sep 16 01:03:32.654971 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.654942 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-workload-certs\") pod \"9b789106-a743-457f-9272-019778117af0\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " Sep 16 01:03:32.654971 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.654968 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b789106-a743-457f-9272-019778117af0-istio-token\") pod \"9b789106-a743-457f-9272-019778117af0\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " Sep 16 01:03:32.655602 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.654994 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b789106-a743-457f-9272-019778117af0-istio-podinfo\") pod \"9b789106-a743-457f-9272-019778117af0\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " Sep 16 01:03:32.655602 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.655151 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-workload-socket\") pod \"9b789106-a743-457f-9272-019778117af0\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " Sep 16 01:03:32.655602 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.655215 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8txm4\" (UniqueName: \"kubernetes.io/projected/9b789106-a743-457f-9272-019778117af0-kube-api-access-8txm4\") pod \"9b789106-a743-457f-9272-019778117af0\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " Sep 16 01:03:32.655602 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.655270 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-credential-socket\") pod \"9b789106-a743-457f-9272-019778117af0\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " Sep 16 01:03:32.655602 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.655314 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-istio-data\") pod \"9b789106-a743-457f-9272-019778117af0\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " Sep 16 01:03:32.655602 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.655316 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9b789106-a743-457f-9272-019778117af0" (UID: "9b789106-a743-457f-9272-019778117af0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:03:32.655602 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.655357 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b789106-a743-457f-9272-019778117af0-istiod-ca-cert\") pod \"9b789106-a743-457f-9272-019778117af0\" (UID: \"9b789106-a743-457f-9272-019778117af0\") " Sep 16 01:03:32.655602 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.655373 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9b789106-a743-457f-9272-019778117af0" (UID: "9b789106-a743-457f-9272-019778117af0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:03:32.655602 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.655528 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9b789106-a743-457f-9272-019778117af0" (UID: "9b789106-a743-457f-9272-019778117af0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:03:32.655965 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.655797 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-istio-data" (OuterVolumeSpecName: "istio-data") pod "9b789106-a743-457f-9272-019778117af0" (UID: "9b789106-a743-457f-9272-019778117af0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:03:32.655965 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.655822 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-workload-certs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:32.655965 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.655847 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-workload-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:32.655965 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.655938 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b789106-a743-457f-9272-019778117af0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9b789106-a743-457f-9272-019778117af0" (UID: "9b789106-a743-457f-9272-019778117af0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:03:32.656376 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.655951 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-credential-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:32.657483 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.657448 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9b789106-a743-457f-9272-019778117af0" (UID: "9b789106-a743-457f-9272-019778117af0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:03:32.657625 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.657503 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b789106-a743-457f-9272-019778117af0-istio-token" (OuterVolumeSpecName: "istio-token") pod "9b789106-a743-457f-9272-019778117af0" (UID: "9b789106-a743-457f-9272-019778117af0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:03:32.657625 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.657579 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b789106-a743-457f-9272-019778117af0-kube-api-access-8txm4" (OuterVolumeSpecName: "kube-api-access-8txm4") pod "9b789106-a743-457f-9272-019778117af0" (UID: "9b789106-a743-457f-9272-019778117af0"). InnerVolumeSpecName "kube-api-access-8txm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:03:32.657774 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.657753 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b789106-a743-457f-9272-019778117af0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9b789106-a743-457f-9272-019778117af0" (UID: "9b789106-a743-457f-9272-019778117af0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 01:03:32.757150 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.757101 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8txm4\" (UniqueName: \"kubernetes.io/projected/9b789106-a743-457f-9272-019778117af0-kube-api-access-8txm4\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:32.757150 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.757135 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-istio-data\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:32.757150 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.757153 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b789106-a743-457f-9272-019778117af0-istiod-ca-cert\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:32.757150 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.757162 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b789106-a743-457f-9272-019778117af0-istio-envoy\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:32.757457 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.757170 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b789106-a743-457f-9272-019778117af0-istio-token\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:32.757457 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:32.757177 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b789106-a743-457f-9272-019778117af0-istio-podinfo\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:33.144809 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:33.144760 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:33.377570 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:33.377542 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" Sep 16 01:03:33.377772 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:33.377542 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf" event={"ID":"9b789106-a743-457f-9272-019778117af0","Type":"ContainerDied","Data":"cb59ed01df361a9848bc82a0fdc3aced7ed02c34fbe5fdf39572f506fab3263b"} Sep 16 01:03:33.377772 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:33.377670 2567 scope.go:117] "RemoveContainer" containerID="b8fb534c5b0d318e37c652a6d9481c4f032bc0ad6ea5f9e4bd4e4fc4b051e868" Sep 16 01:03:33.402360 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:33.402271 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf"] Sep 16 01:03:33.406403 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:33.406364 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2c31c5-openshift-default-78d474674b-lgvjf"] Sep 16 01:03:34.120287 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:34.120245 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b789106-a743-457f-9272-019778117af0" path="/var/lib/kubelet/pods/9b789106-a743-457f-9272-019778117af0/volumes" Sep 16 01:03:35.599399 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:35.599367 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:36.627263 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:36.627228 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:37.530775 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.530738 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:37.668630 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.668542 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:03:37.807189 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.807152 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:37.810252 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.810223 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-podinfo\") pod \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " Sep 16 01:03:37.810344 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.810267 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-workload-certs\") pod \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " Sep 16 01:03:37.810344 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.810290 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-envoy\") pod \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " Sep 16 01:03:37.810344 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.810331 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-workload-socket\") pod \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " Sep 16 01:03:37.810487 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.810348 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-credential-socket\") pod \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " Sep 16 01:03:37.810487 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.810422 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n97d9\" (UniqueName: \"kubernetes.io/projected/1aba147f-9352-4b2b-9e23-7eac729a8fd8-kube-api-access-n97d9\") pod \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " Sep 16 01:03:37.810487 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.810453 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-token\") pod \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " Sep 16 01:03:37.810649 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.810512 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istiod-ca-cert\") pod \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " Sep 16 01:03:37.810649 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.810553 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-data\") pod \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\" (UID: \"1aba147f-9352-4b2b-9e23-7eac729a8fd8\") " Sep 16 01:03:37.810764 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.810654 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1aba147f-9352-4b2b-9e23-7eac729a8fd8" (UID: "1aba147f-9352-4b2b-9e23-7eac729a8fd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:03:37.810764 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.810683 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1aba147f-9352-4b2b-9e23-7eac729a8fd8" (UID: "1aba147f-9352-4b2b-9e23-7eac729a8fd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:03:37.810869 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.810851 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-data" (OuterVolumeSpecName: "istio-data") pod "1aba147f-9352-4b2b-9e23-7eac729a8fd8" (UID: "1aba147f-9352-4b2b-9e23-7eac729a8fd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:03:37.810987 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.810958 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1aba147f-9352-4b2b-9e23-7eac729a8fd8" (UID: "1aba147f-9352-4b2b-9e23-7eac729a8fd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:03:37.810987 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.810964 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1aba147f-9352-4b2b-9e23-7eac729a8fd8" (UID: "1aba147f-9352-4b2b-9e23-7eac729a8fd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:03:37.811364 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.810989 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-workload-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:37.811364 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.811012 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-data\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:37.811364 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.811026 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-workload-certs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:37.812873 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.812844 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1aba147f-9352-4b2b-9e23-7eac729a8fd8" (UID: "1aba147f-9352-4b2b-9e23-7eac729a8fd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 01:03:37.812988 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.812900 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-token" (OuterVolumeSpecName: "istio-token") pod "1aba147f-9352-4b2b-9e23-7eac729a8fd8" (UID: "1aba147f-9352-4b2b-9e23-7eac729a8fd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:03:37.812988 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.812921 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1aba147f-9352-4b2b-9e23-7eac729a8fd8" (UID: "1aba147f-9352-4b2b-9e23-7eac729a8fd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:03:37.813104 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.812978 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aba147f-9352-4b2b-9e23-7eac729a8fd8-kube-api-access-n97d9" (OuterVolumeSpecName: "kube-api-access-n97d9") pod "1aba147f-9352-4b2b-9e23-7eac729a8fd8" (UID: "1aba147f-9352-4b2b-9e23-7eac729a8fd8"). InnerVolumeSpecName "kube-api-access-n97d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:03:37.912486 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.912445 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n97d9\" (UniqueName: \"kubernetes.io/projected/1aba147f-9352-4b2b-9e23-7eac729a8fd8-kube-api-access-n97d9\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:37.912486 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.912479 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-token\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:37.912486 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.912490 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istiod-ca-cert\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:37.912486 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.912498 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-podinfo\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:37.912795 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.912506 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-istio-envoy\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:37.912795 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:37.912514 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aba147f-9352-4b2b-9e23-7eac729a8fd8-credential-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:38.295659 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:38.295617 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:38.411397 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:38.411286 2567 generic.go:356] "Generic (PLEG): container finished" podID="1aba147f-9352-4b2b-9e23-7eac729a8fd8" containerID="51f4b9b8c8294bf14e7851abb7c9ca7868e1e8a9fe924f311c3791ac7cc8f06a" exitCode=0 Sep 16 01:03:38.411397 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:38.411375 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" event={"ID":"1aba147f-9352-4b2b-9e23-7eac729a8fd8","Type":"ContainerDied","Data":"51f4b9b8c8294bf14e7851abb7c9ca7868e1e8a9fe924f311c3791ac7cc8f06a"} Sep 16 01:03:38.411627 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:38.411381 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" Sep 16 01:03:38.411627 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:38.411430 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994" event={"ID":"1aba147f-9352-4b2b-9e23-7eac729a8fd8","Type":"ContainerDied","Data":"8ca65719ea45fc8d12de64aeb94eb90207ae92b7f117cec94dacb2c3fb1a06ef"} Sep 16 01:03:38.411627 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:38.411454 2567 scope.go:117] "RemoveContainer" containerID="51f4b9b8c8294bf14e7851abb7c9ca7868e1e8a9fe924f311c3791ac7cc8f06a" Sep 16 01:03:38.421898 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:38.421879 2567 scope.go:117] "RemoveContainer" containerID="211ef5dcb6870bef843fb4d5c48ba9660fdc2a1b67793aacd55f5edb62034289" Sep 16 01:03:38.431313 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:38.431273 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994"] Sep 16 01:03:38.433743 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:38.433718 2567 scope.go:117] "RemoveContainer" containerID="51f4b9b8c8294bf14e7851abb7c9ca7868e1e8a9fe924f311c3791ac7cc8f06a" Sep 16 01:03:38.434174 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:03:38.434154 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51f4b9b8c8294bf14e7851abb7c9ca7868e1e8a9fe924f311c3791ac7cc8f06a\": container with ID starting with 51f4b9b8c8294bf14e7851abb7c9ca7868e1e8a9fe924f311c3791ac7cc8f06a not found: ID does not exist" containerID="51f4b9b8c8294bf14e7851abb7c9ca7868e1e8a9fe924f311c3791ac7cc8f06a" Sep 16 01:03:38.434228 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:38.434184 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51f4b9b8c8294bf14e7851abb7c9ca7868e1e8a9fe924f311c3791ac7cc8f06a"} err="failed to get container status \"51f4b9b8c8294bf14e7851abb7c9ca7868e1e8a9fe924f311c3791ac7cc8f06a\": rpc error: code = NotFound desc = could not find container \"51f4b9b8c8294bf14e7851abb7c9ca7868e1e8a9fe924f311c3791ac7cc8f06a\": container with ID starting with 51f4b9b8c8294bf14e7851abb7c9ca7868e1e8a9fe924f311c3791ac7cc8f06a not found: ID does not exist" Sep 16 01:03:38.434228 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:38.434203 2567 scope.go:117] "RemoveContainer" containerID="211ef5dcb6870bef843fb4d5c48ba9660fdc2a1b67793aacd55f5edb62034289" Sep 16 01:03:38.434494 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:03:38.434478 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211ef5dcb6870bef843fb4d5c48ba9660fdc2a1b67793aacd55f5edb62034289\": container with ID starting with 211ef5dcb6870bef843fb4d5c48ba9660fdc2a1b67793aacd55f5edb62034289 not found: ID does not exist" containerID="211ef5dcb6870bef843fb4d5c48ba9660fdc2a1b67793aacd55f5edb62034289" Sep 16 01:03:38.434533 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:38.434498 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211ef5dcb6870bef843fb4d5c48ba9660fdc2a1b67793aacd55f5edb62034289"} err="failed to get container status \"211ef5dcb6870bef843fb4d5c48ba9660fdc2a1b67793aacd55f5edb62034289\": rpc error: code = NotFound desc = could not find container \"211ef5dcb6870bef843fb4d5c48ba9660fdc2a1b67793aacd55f5edb62034289\": container with ID starting with 211ef5dcb6870bef843fb4d5c48ba9660fdc2a1b67793aacd55f5edb62034289 not found: ID does not exist" Sep 16 01:03:38.435449 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:38.435430 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eba3d3-openshift-default-5b744d68cb-vb994"] Sep 16 01:03:39.033718 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:39.033667 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:39.697647 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:39.697614 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:40.120614 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:40.120577 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aba147f-9352-4b2b-9e23-7eac729a8fd8" path="/var/lib/kubelet/pods/1aba147f-9352-4b2b-9e23-7eac729a8fd8/volumes" Sep 16 01:03:40.436939 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:40.436846 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:41.230923 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:41.230882 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:44.604933 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:44.604895 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:44.672549 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:44.672474 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:44.771564 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:44.771532 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:44.828567 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:44.828513 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:45.245934 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:45.245896 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:45.343687 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:45.343648 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:45.969791 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:45.969753 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:46.824418 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:46.824385 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:47.695523 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:47.695481 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:48.537423 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:48.537340 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:49.203645 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:49.203607 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:50.238423 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:50.238384 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:03:50.886712 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:50.886668 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:50.947290 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:50.947259 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:51.136472 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:51.136432 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n"] Sep 16 01:03:51.136806 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:51.136736 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" podUID="124e2e72-5d99-4745-92a0-a6624da3ea07" containerName="istio-proxy" containerID="cri-o://50de2a8d95d58a99b28e8ab6e2c398c7b933e38c9926cb06a29f741493f36e73" gracePeriod=30 Sep 16 01:03:51.152868 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:51.152836 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:51.823807 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:51.823764 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:52.570561 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:52.570519 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:52.601802 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:52.601765 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55"] Sep 16 01:03:52.602107 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:52.602070 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55" podUID="21cb4d08-16e2-4d5a-9f9e-59c394282a26" containerName="payload-processing" containerID="cri-o://86ab8533f4e80e6ebcb4d9e7eeec2d855b1c86c96a319c02a0fca753d0731c82" gracePeriod=30 Sep 16 01:03:52.670591 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:52.670556 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:52.756679 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:52.756630 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:52.824556 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:52.824472 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:52.879833 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:52.879804 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55" Sep 16 01:03:52.951543 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:52.951502 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:52.966609 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:52.966575 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg9jp\" (UniqueName: \"kubernetes.io/projected/21cb4d08-16e2-4d5a-9f9e-59c394282a26-kube-api-access-tg9jp\") pod \"21cb4d08-16e2-4d5a-9f9e-59c394282a26\" (UID: \"21cb4d08-16e2-4d5a-9f9e-59c394282a26\") " Sep 16 01:03:52.966840 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:52.966691 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21cb4d08-16e2-4d5a-9f9e-59c394282a26-plugins-config-volume\") pod \"21cb4d08-16e2-4d5a-9f9e-59c394282a26\" (UID: \"21cb4d08-16e2-4d5a-9f9e-59c394282a26\") " Sep 16 01:03:52.967135 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:52.967105 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21cb4d08-16e2-4d5a-9f9e-59c394282a26-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "21cb4d08-16e2-4d5a-9f9e-59c394282a26" (UID: "21cb4d08-16e2-4d5a-9f9e-59c394282a26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:03:52.969464 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:52.969432 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21cb4d08-16e2-4d5a-9f9e-59c394282a26-kube-api-access-tg9jp" (OuterVolumeSpecName: "kube-api-access-tg9jp") pod "21cb4d08-16e2-4d5a-9f9e-59c394282a26" (UID: "21cb4d08-16e2-4d5a-9f9e-59c394282a26"). InnerVolumeSpecName "kube-api-access-tg9jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:03:53.068300 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:53.068262 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg9jp\" (UniqueName: \"kubernetes.io/projected/21cb4d08-16e2-4d5a-9f9e-59c394282a26-kube-api-access-tg9jp\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:53.068300 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:53.068293 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21cb4d08-16e2-4d5a-9f9e-59c394282a26-plugins-config-volume\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:53.172742 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:53.172641 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:53.519698 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:53.519595 2567 generic.go:356] "Generic (PLEG): container finished" podID="21cb4d08-16e2-4d5a-9f9e-59c394282a26" containerID="86ab8533f4e80e6ebcb4d9e7eeec2d855b1c86c96a319c02a0fca753d0731c82" exitCode=0 Sep 16 01:03:53.519698 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:53.519644 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55" Sep 16 01:03:53.519698 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:53.519684 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55" event={"ID":"21cb4d08-16e2-4d5a-9f9e-59c394282a26","Type":"ContainerDied","Data":"86ab8533f4e80e6ebcb4d9e7eeec2d855b1c86c96a319c02a0fca753d0731c82"} Sep 16 01:03:53.519972 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:53.519745 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55" event={"ID":"21cb4d08-16e2-4d5a-9f9e-59c394282a26","Type":"ContainerDied","Data":"545ec4b8dc338d278ddb05f89c1e833fa2359c30b5a22a01e24636e1252d339b"} Sep 16 01:03:53.519972 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:53.519775 2567 scope.go:117] "RemoveContainer" containerID="86ab8533f4e80e6ebcb4d9e7eeec2d855b1c86c96a319c02a0fca753d0731c82" Sep 16 01:03:53.530388 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:53.530362 2567 scope.go:117] "RemoveContainer" containerID="86ab8533f4e80e6ebcb4d9e7eeec2d855b1c86c96a319c02a0fca753d0731c82" Sep 16 01:03:53.530733 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:03:53.530708 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ab8533f4e80e6ebcb4d9e7eeec2d855b1c86c96a319c02a0fca753d0731c82\": container with ID starting with 86ab8533f4e80e6ebcb4d9e7eeec2d855b1c86c96a319c02a0fca753d0731c82 not found: ID does not exist" containerID="86ab8533f4e80e6ebcb4d9e7eeec2d855b1c86c96a319c02a0fca753d0731c82" Sep 16 01:03:53.530802 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:53.530745 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ab8533f4e80e6ebcb4d9e7eeec2d855b1c86c96a319c02a0fca753d0731c82"} err="failed to get container status \"86ab8533f4e80e6ebcb4d9e7eeec2d855b1c86c96a319c02a0fca753d0731c82\": rpc error: code = NotFound desc = could not find container \"86ab8533f4e80e6ebcb4d9e7eeec2d855b1c86c96a319c02a0fca753d0731c82\": container with ID starting with 86ab8533f4e80e6ebcb4d9e7eeec2d855b1c86c96a319c02a0fca753d0731c82 not found: ID does not exist" Sep 16 01:03:53.542952 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:53.542917 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55"] Sep 16 01:03:53.549167 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:53.549122 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-715834-846849cbcc-b7n55"] Sep 16 01:03:53.723869 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:53.723820 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:54.119053 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:54.119012 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21cb4d08-16e2-4d5a-9f9e-59c394282a26" path="/var/lib/kubelet/pods/21cb4d08-16e2-4d5a-9f9e-59c394282a26/volumes" Sep 16 01:03:54.415032 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:54.414939 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:55.353257 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:55.353219 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:56.047308 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.047273 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:56.437847 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.437819 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:03:56.506304 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.506260 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-token\") pod \"124e2e72-5d99-4745-92a0-a6624da3ea07\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " Sep 16 01:03:56.506492 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.506321 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-envoy\") pod \"124e2e72-5d99-4745-92a0-a6624da3ea07\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " Sep 16 01:03:56.506492 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.506372 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/124e2e72-5d99-4745-92a0-a6624da3ea07-istiod-ca-cert\") pod \"124e2e72-5d99-4745-92a0-a6624da3ea07\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " Sep 16 01:03:56.506492 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.506397 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-workload-socket\") pod \"124e2e72-5d99-4745-92a0-a6624da3ea07\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " Sep 16 01:03:56.506492 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.506459 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-workload-certs\") pod \"124e2e72-5d99-4745-92a0-a6624da3ea07\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " Sep 16 01:03:56.506492 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.506474 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-credential-socket\") pod \"124e2e72-5d99-4745-92a0-a6624da3ea07\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " Sep 16 01:03:56.506751 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.506537 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5kh2\" (UniqueName: \"kubernetes.io/projected/124e2e72-5d99-4745-92a0-a6624da3ea07-kube-api-access-t5kh2\") pod \"124e2e72-5d99-4745-92a0-a6624da3ea07\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " Sep 16 01:03:56.506751 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.506560 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-data\") pod \"124e2e72-5d99-4745-92a0-a6624da3ea07\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " Sep 16 01:03:56.506751 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.506603 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-podinfo\") pod \"124e2e72-5d99-4745-92a0-a6624da3ea07\" (UID: \"124e2e72-5d99-4745-92a0-a6624da3ea07\") " Sep 16 01:03:56.506885 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.506769 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/124e2e72-5d99-4745-92a0-a6624da3ea07-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "124e2e72-5d99-4745-92a0-a6624da3ea07" (UID: "124e2e72-5d99-4745-92a0-a6624da3ea07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:03:56.506885 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.506772 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "124e2e72-5d99-4745-92a0-a6624da3ea07" (UID: "124e2e72-5d99-4745-92a0-a6624da3ea07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:03:56.506885 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.506777 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "124e2e72-5d99-4745-92a0-a6624da3ea07" (UID: "124e2e72-5d99-4745-92a0-a6624da3ea07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:03:56.507029 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.506996 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-data" (OuterVolumeSpecName: "istio-data") pod "124e2e72-5d99-4745-92a0-a6624da3ea07" (UID: "124e2e72-5d99-4745-92a0-a6624da3ea07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:03:56.507029 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.507004 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/124e2e72-5d99-4745-92a0-a6624da3ea07-istiod-ca-cert\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:56.507211 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.507085 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-workload-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:56.507211 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.507105 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-credential-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:56.507211 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.507124 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "124e2e72-5d99-4745-92a0-a6624da3ea07" (UID: "124e2e72-5d99-4745-92a0-a6624da3ea07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:03:56.508911 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.508878 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "124e2e72-5d99-4745-92a0-a6624da3ea07" (UID: "124e2e72-5d99-4745-92a0-a6624da3ea07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 01:03:56.509089 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.508917 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-token" (OuterVolumeSpecName: "istio-token") pod "124e2e72-5d99-4745-92a0-a6624da3ea07" (UID: "124e2e72-5d99-4745-92a0-a6624da3ea07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:03:56.509089 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.508976 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "124e2e72-5d99-4745-92a0-a6624da3ea07" (UID: "124e2e72-5d99-4745-92a0-a6624da3ea07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:03:56.509089 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.509000 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124e2e72-5d99-4745-92a0-a6624da3ea07-kube-api-access-t5kh2" (OuterVolumeSpecName: "kube-api-access-t5kh2") pod "124e2e72-5d99-4745-92a0-a6624da3ea07" (UID: "124e2e72-5d99-4745-92a0-a6624da3ea07"). InnerVolumeSpecName "kube-api-access-t5kh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:03:56.541559 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.541518 2567 generic.go:356] "Generic (PLEG): container finished" podID="124e2e72-5d99-4745-92a0-a6624da3ea07" containerID="50de2a8d95d58a99b28e8ab6e2c398c7b933e38c9926cb06a29f741493f36e73" exitCode=0 Sep 16 01:03:56.541753 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.541570 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" Sep 16 01:03:56.541753 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.541601 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" event={"ID":"124e2e72-5d99-4745-92a0-a6624da3ea07","Type":"ContainerDied","Data":"50de2a8d95d58a99b28e8ab6e2c398c7b933e38c9926cb06a29f741493f36e73"} Sep 16 01:03:56.541753 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.541658 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n" event={"ID":"124e2e72-5d99-4745-92a0-a6624da3ea07","Type":"ContainerDied","Data":"00790d69ac0c0d92b6bf8d207022f3ee639f0c6569e5b84b3a154f66b1131a75"} Sep 16 01:03:56.541753 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.541684 2567 scope.go:117] "RemoveContainer" containerID="50de2a8d95d58a99b28e8ab6e2c398c7b933e38c9926cb06a29f741493f36e73" Sep 16 01:03:56.552132 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.552112 2567 scope.go:117] "RemoveContainer" containerID="6c471907d5e14eddf605efad0d7438bcaece6be2937da7bd3e5394e5af82d6ba" Sep 16 01:03:56.562895 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.562870 2567 scope.go:117] "RemoveContainer" containerID="50de2a8d95d58a99b28e8ab6e2c398c7b933e38c9926cb06a29f741493f36e73" Sep 16 01:03:56.563307 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:03:56.563282 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50de2a8d95d58a99b28e8ab6e2c398c7b933e38c9926cb06a29f741493f36e73\": container with ID starting with 50de2a8d95d58a99b28e8ab6e2c398c7b933e38c9926cb06a29f741493f36e73 not found: ID does not exist" containerID="50de2a8d95d58a99b28e8ab6e2c398c7b933e38c9926cb06a29f741493f36e73" Sep 16 01:03:56.563413 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.563320 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50de2a8d95d58a99b28e8ab6e2c398c7b933e38c9926cb06a29f741493f36e73"} err="failed to get container status \"50de2a8d95d58a99b28e8ab6e2c398c7b933e38c9926cb06a29f741493f36e73\": rpc error: code = NotFound desc = could not find container \"50de2a8d95d58a99b28e8ab6e2c398c7b933e38c9926cb06a29f741493f36e73\": container with ID starting with 50de2a8d95d58a99b28e8ab6e2c398c7b933e38c9926cb06a29f741493f36e73 not found: ID does not exist" Sep 16 01:03:56.563413 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.563346 2567 scope.go:117] "RemoveContainer" containerID="6c471907d5e14eddf605efad0d7438bcaece6be2937da7bd3e5394e5af82d6ba" Sep 16 01:03:56.563637 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:03:56.563617 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c471907d5e14eddf605efad0d7438bcaece6be2937da7bd3e5394e5af82d6ba\": container with ID starting with 6c471907d5e14eddf605efad0d7438bcaece6be2937da7bd3e5394e5af82d6ba not found: ID does not exist" containerID="6c471907d5e14eddf605efad0d7438bcaece6be2937da7bd3e5394e5af82d6ba" Sep 16 01:03:56.563677 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.563647 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c471907d5e14eddf605efad0d7438bcaece6be2937da7bd3e5394e5af82d6ba"} err="failed to get container status \"6c471907d5e14eddf605efad0d7438bcaece6be2937da7bd3e5394e5af82d6ba\": rpc error: code = NotFound desc = could not find container \"6c471907d5e14eddf605efad0d7438bcaece6be2937da7bd3e5394e5af82d6ba\": container with ID starting with 6c471907d5e14eddf605efad0d7438bcaece6be2937da7bd3e5394e5af82d6ba not found: ID does not exist" Sep 16 01:03:56.564965 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.564908 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n"] Sep 16 01:03:56.568163 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.568086 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-761551-openshift-default-54d7857944-bg55n"] Sep 16 01:03:56.608437 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.608399 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-workload-certs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:56.608437 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.608432 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5kh2\" (UniqueName: \"kubernetes.io/projected/124e2e72-5d99-4745-92a0-a6624da3ea07-kube-api-access-t5kh2\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:56.608437 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.608444 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-data\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:56.608668 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.608455 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-podinfo\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:56.608668 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.608463 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-token\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:56.608668 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.608471 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/124e2e72-5d99-4745-92a0-a6624da3ea07-istio-envoy\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:03:56.821455 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:56.821364 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:03:58.119959 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:58.119922 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="124e2e72-5d99-4745-92a0-a6624da3ea07" path="/var/lib/kubelet/pods/124e2e72-5d99-4745-92a0-a6624da3ea07/volumes" Sep 16 01:03:58.149488 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:58.149448 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp"] Sep 16 01:03:58.187270 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:03:58.187234 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:01.681764 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:01.681682 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:02.266922 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:02.266881 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3462ea-29825340-tvwrb"] Sep 16 01:04:02.270820 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:02.270784 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:02.272314 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:02.272289 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3462ea-29825340-tvwrb"] Sep 16 01:04:02.370679 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:02.370638 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:02.469651 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:02.469621 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:02.529889 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:02.529788 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:02.831778 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:02.831744 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:02.942622 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:02.942590 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:03.736911 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:03.736879 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:04.119914 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:04.119879 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24325c82-c840-4ea5-8df5-dc873e840b57" path="/var/lib/kubelet/pods/24325c82-c840-4ea5-8df5-dc873e840b57/volumes" Sep 16 01:04:04.472219 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:04.472126 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:05.227802 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:05.227759 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:06.642789 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:06.642731 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:04:08.488939 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:08.488844 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:08.588755 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:08.588717 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:08.839650 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:08.839615 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:09.242644 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:09.242542 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:09.932395 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:09.932361 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:10.846376 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:10.846337 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:11.527607 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:11.527570 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:12.225407 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:12.225375 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:13.171735 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.171687 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" podUID="cfcd3554-65e7-45c1-94f5-df99995fead2" containerName="main" containerID="cri-o://4339494f3d697fd10f3aaa549d14cf5e1c74979452ff25a4836a3ec0a1e45378" gracePeriod=45 Sep 16 01:04:13.434475 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.434403 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:04:13.482898 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.482852 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-tmp-dir\") pod \"cfcd3554-65e7-45c1-94f5-df99995fead2\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " Sep 16 01:04:13.483130 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.482920 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lq8n\" (UniqueName: \"kubernetes.io/projected/cfcd3554-65e7-45c1-94f5-df99995fead2-kube-api-access-5lq8n\") pod \"cfcd3554-65e7-45c1-94f5-df99995fead2\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " Sep 16 01:04:13.483130 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.483009 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-model-cache\") pod \"cfcd3554-65e7-45c1-94f5-df99995fead2\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " Sep 16 01:04:13.483130 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.483054 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-dshm\") pod \"cfcd3554-65e7-45c1-94f5-df99995fead2\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " Sep 16 01:04:13.483130 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.483083 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-home\") pod \"cfcd3554-65e7-45c1-94f5-df99995fead2\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " Sep 16 01:04:13.483362 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.483209 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfcd3554-65e7-45c1-94f5-df99995fead2" (UID: "cfcd3554-65e7-45c1-94f5-df99995fead2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:04:13.483422 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.483363 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfcd3554-65e7-45c1-94f5-df99995fead2" (UID: "cfcd3554-65e7-45c1-94f5-df99995fead2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:04:13.483484 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.483456 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-home" (OuterVolumeSpecName: "home") pod "cfcd3554-65e7-45c1-94f5-df99995fead2" (UID: "cfcd3554-65e7-45c1-94f5-df99995fead2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:04:13.483534 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.483481 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-model-cache\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:04:13.483534 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.483502 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-tmp-dir\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:04:13.485640 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.485576 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfcd3554-65e7-45c1-94f5-df99995fead2-kube-api-access-5lq8n" (OuterVolumeSpecName: "kube-api-access-5lq8n") pod "cfcd3554-65e7-45c1-94f5-df99995fead2" (UID: "cfcd3554-65e7-45c1-94f5-df99995fead2"). InnerVolumeSpecName "kube-api-access-5lq8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:04:13.485866 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.485844 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-dshm" (OuterVolumeSpecName: "dshm") pod "cfcd3554-65e7-45c1-94f5-df99995fead2" (UID: "cfcd3554-65e7-45c1-94f5-df99995fead2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:04:13.584380 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.584340 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfcd3554-65e7-45c1-94f5-df99995fead2-tls-certs\") pod \"cfcd3554-65e7-45c1-94f5-df99995fead2\" (UID: \"cfcd3554-65e7-45c1-94f5-df99995fead2\") " Sep 16 01:04:13.584619 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.584602 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-dshm\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:04:13.584674 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.584624 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfcd3554-65e7-45c1-94f5-df99995fead2-home\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:04:13.584674 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.584634 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lq8n\" (UniqueName: \"kubernetes.io/projected/cfcd3554-65e7-45c1-94f5-df99995fead2-kube-api-access-5lq8n\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:04:13.586772 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.586742 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfcd3554-65e7-45c1-94f5-df99995fead2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfcd3554-65e7-45c1-94f5-df99995fead2" (UID: "cfcd3554-65e7-45c1-94f5-df99995fead2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:04:13.648048 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.648006 2567 generic.go:356] "Generic (PLEG): container finished" podID="cfcd3554-65e7-45c1-94f5-df99995fead2" containerID="4339494f3d697fd10f3aaa549d14cf5e1c74979452ff25a4836a3ec0a1e45378" exitCode=0 Sep 16 01:04:13.648242 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.648080 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" Sep 16 01:04:13.648242 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.648103 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" event={"ID":"cfcd3554-65e7-45c1-94f5-df99995fead2","Type":"ContainerDied","Data":"4339494f3d697fd10f3aaa549d14cf5e1c74979452ff25a4836a3ec0a1e45378"} Sep 16 01:04:13.648242 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.648153 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp" event={"ID":"cfcd3554-65e7-45c1-94f5-df99995fead2","Type":"ContainerDied","Data":"87a824332c98dd5bfed6e1823af3a7f8c173a63869142fe58ab6174db05723e5"} Sep 16 01:04:13.648242 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.648176 2567 scope.go:117] "RemoveContainer" containerID="4339494f3d697fd10f3aaa549d14cf5e1c74979452ff25a4836a3ec0a1e45378" Sep 16 01:04:13.658474 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.658453 2567 scope.go:117] "RemoveContainer" containerID="4339494f3d697fd10f3aaa549d14cf5e1c74979452ff25a4836a3ec0a1e45378" Sep 16 01:04:13.658827 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:04:13.658802 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4339494f3d697fd10f3aaa549d14cf5e1c74979452ff25a4836a3ec0a1e45378\": container with ID starting with 4339494f3d697fd10f3aaa549d14cf5e1c74979452ff25a4836a3ec0a1e45378 not found: ID does not exist" containerID="4339494f3d697fd10f3aaa549d14cf5e1c74979452ff25a4836a3ec0a1e45378" Sep 16 01:04:13.658918 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.658834 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4339494f3d697fd10f3aaa549d14cf5e1c74979452ff25a4836a3ec0a1e45378"} err="failed to get container status \"4339494f3d697fd10f3aaa549d14cf5e1c74979452ff25a4836a3ec0a1e45378\": rpc error: code = NotFound desc = could not find container \"4339494f3d697fd10f3aaa549d14cf5e1c74979452ff25a4836a3ec0a1e45378\": container with ID starting with 4339494f3d697fd10f3aaa549d14cf5e1c74979452ff25a4836a3ec0a1e45378 not found: ID does not exist" Sep 16 01:04:13.670762 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.670730 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp"] Sep 16 01:04:13.674648 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.674616 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-715834/ipp-route-715834-kserve-6b4b598d59-6tbhp"] Sep 16 01:04:13.685208 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:13.685113 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfcd3554-65e7-45c1-94f5-df99995fead2-tls-certs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:04:14.120274 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:14.120237 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfcd3554-65e7-45c1-94f5-df99995fead2" path="/var/lib/kubelet/pods/cfcd3554-65e7-45c1-94f5-df99995fead2/volumes" Sep 16 01:04:19.659202 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:19.659156 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:19.727066 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:19.726996 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:19.824188 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:19.824148 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:19.923457 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:19.923299 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:20.215881 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:20.215735 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:20.316407 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:20.316371 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:21.009522 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:21.009488 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:21.831012 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:21.830979 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:23.840923 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:23.840884 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:04:26.055825 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:26.055789 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:26.153467 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:26.153428 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:26.299300 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:26.299262 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l"] Sep 16 01:04:26.299598 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:26.299566 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" podUID="177f8426-32c4-43a9-b689-7984a4b2cd0f" containerName="istio-proxy" containerID="cri-o://9d50fbd075c513bcf16d120047ea5b37ead86c790eeabc53a8de2b7e9f034a6d" gracePeriod=30 Sep 16 01:04:26.314604 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:26.314513 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:27.986819 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:27.986777 2567 patch_prober.go:28] interesting pod/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 01:04:27.987232 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:27.986864 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" podUID="177f8426-32c4-43a9-b689-7984a4b2cd0f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 01:04:31.558571 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.558540 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 01:04:31.656544 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.656442 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-workload-certs\") pod \"177f8426-32c4-43a9-b689-7984a4b2cd0f\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " Sep 16 01:04:31.656544 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.656507 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-token\") pod \"177f8426-32c4-43a9-b689-7984a4b2cd0f\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " Sep 16 01:04:31.656544 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.656546 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-envoy\") pod \"177f8426-32c4-43a9-b689-7984a4b2cd0f\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " Sep 16 01:04:31.656855 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.656573 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8bvv\" (UniqueName: \"kubernetes.io/projected/177f8426-32c4-43a9-b689-7984a4b2cd0f-kube-api-access-n8bvv\") pod \"177f8426-32c4-43a9-b689-7984a4b2cd0f\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " Sep 16 01:04:31.656855 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.656617 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177f8426-32c4-43a9-b689-7984a4b2cd0f-istiod-ca-cert\") pod \"177f8426-32c4-43a9-b689-7984a4b2cd0f\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " Sep 16 01:04:31.656855 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.656640 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-workload-socket\") pod \"177f8426-32c4-43a9-b689-7984a4b2cd0f\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " Sep 16 01:04:31.656855 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.656684 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-credential-socket\") pod \"177f8426-32c4-43a9-b689-7984a4b2cd0f\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " Sep 16 01:04:31.656855 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.656717 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-data\") pod \"177f8426-32c4-43a9-b689-7984a4b2cd0f\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " Sep 16 01:04:31.656855 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.656744 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "177f8426-32c4-43a9-b689-7984a4b2cd0f" (UID: "177f8426-32c4-43a9-b689-7984a4b2cd0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:04:31.656855 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.656767 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-podinfo\") pod \"177f8426-32c4-43a9-b689-7984a4b2cd0f\" (UID: \"177f8426-32c4-43a9-b689-7984a4b2cd0f\") " Sep 16 01:04:31.657373 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.657344 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-data" (OuterVolumeSpecName: "istio-data") pod "177f8426-32c4-43a9-b689-7984a4b2cd0f" (UID: "177f8426-32c4-43a9-b689-7984a4b2cd0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:04:31.657564 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.656917 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "177f8426-32c4-43a9-b689-7984a4b2cd0f" (UID: "177f8426-32c4-43a9-b689-7984a4b2cd0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:04:31.657564 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.657517 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "177f8426-32c4-43a9-b689-7984a4b2cd0f" (UID: "177f8426-32c4-43a9-b689-7984a4b2cd0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:04:31.659486 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.658128 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-workload-certs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:04:31.659486 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.658279 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/177f8426-32c4-43a9-b689-7984a4b2cd0f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "177f8426-32c4-43a9-b689-7984a4b2cd0f" (UID: "177f8426-32c4-43a9-b689-7984a4b2cd0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:04:31.660229 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.660093 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/177f8426-32c4-43a9-b689-7984a4b2cd0f-kube-api-access-n8bvv" (OuterVolumeSpecName: "kube-api-access-n8bvv") pod "177f8426-32c4-43a9-b689-7984a4b2cd0f" (UID: "177f8426-32c4-43a9-b689-7984a4b2cd0f"). InnerVolumeSpecName "kube-api-access-n8bvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:04:31.660374 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.660345 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "177f8426-32c4-43a9-b689-7984a4b2cd0f" (UID: "177f8426-32c4-43a9-b689-7984a4b2cd0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 01:04:31.666535 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.666498 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-token" (OuterVolumeSpecName: "istio-token") pod "177f8426-32c4-43a9-b689-7984a4b2cd0f" (UID: "177f8426-32c4-43a9-b689-7984a4b2cd0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:04:31.666839 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.666812 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "177f8426-32c4-43a9-b689-7984a4b2cd0f" (UID: "177f8426-32c4-43a9-b689-7984a4b2cd0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:04:31.758873 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.758844 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-token\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:04:31.758873 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.758869 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-envoy\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:04:31.758873 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.758878 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8bvv\" (UniqueName: \"kubernetes.io/projected/177f8426-32c4-43a9-b689-7984a4b2cd0f-kube-api-access-n8bvv\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:04:31.759206 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.758888 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177f8426-32c4-43a9-b689-7984a4b2cd0f-istiod-ca-cert\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:04:31.759206 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.758897 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-workload-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:04:31.759206 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.758907 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-credential-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:04:31.759206 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.758915 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-data\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:04:31.759206 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.758924 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177f8426-32c4-43a9-b689-7984a4b2cd0f-istio-podinfo\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:04:31.760686 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.760656 2567 generic.go:356] "Generic (PLEG): container finished" podID="177f8426-32c4-43a9-b689-7984a4b2cd0f" containerID="9d50fbd075c513bcf16d120047ea5b37ead86c790eeabc53a8de2b7e9f034a6d" exitCode=0 Sep 16 01:04:31.760808 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.760726 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" Sep 16 01:04:31.760808 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.760737 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" event={"ID":"177f8426-32c4-43a9-b689-7984a4b2cd0f","Type":"ContainerDied","Data":"9d50fbd075c513bcf16d120047ea5b37ead86c790eeabc53a8de2b7e9f034a6d"} Sep 16 01:04:31.760880 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.760819 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l" event={"ID":"177f8426-32c4-43a9-b689-7984a4b2cd0f","Type":"ContainerDied","Data":"b815ea5da96a78dbc4188f742bce81e1cba3cd4ef8f8b30c202ecc10eb433111"} Sep 16 01:04:31.760880 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.760841 2567 scope.go:117] "RemoveContainer" containerID="9d50fbd075c513bcf16d120047ea5b37ead86c790eeabc53a8de2b7e9f034a6d" Sep 16 01:04:31.771593 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.771571 2567 scope.go:117] "RemoveContainer" containerID="735d94fd1533eba001fec7b939d8de647c4cc95b278cb6b1bd77e39467ca4255" Sep 16 01:04:31.782118 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.781828 2567 scope.go:117] "RemoveContainer" containerID="9d50fbd075c513bcf16d120047ea5b37ead86c790eeabc53a8de2b7e9f034a6d" Sep 16 01:04:31.782486 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:04:31.782454 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d50fbd075c513bcf16d120047ea5b37ead86c790eeabc53a8de2b7e9f034a6d\": container with ID starting with 9d50fbd075c513bcf16d120047ea5b37ead86c790eeabc53a8de2b7e9f034a6d not found: ID does not exist" containerID="9d50fbd075c513bcf16d120047ea5b37ead86c790eeabc53a8de2b7e9f034a6d" Sep 16 01:04:31.782598 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.782499 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d50fbd075c513bcf16d120047ea5b37ead86c790eeabc53a8de2b7e9f034a6d"} err="failed to get container status \"9d50fbd075c513bcf16d120047ea5b37ead86c790eeabc53a8de2b7e9f034a6d\": rpc error: code = NotFound desc = could not find container \"9d50fbd075c513bcf16d120047ea5b37ead86c790eeabc53a8de2b7e9f034a6d\": container with ID starting with 9d50fbd075c513bcf16d120047ea5b37ead86c790eeabc53a8de2b7e9f034a6d not found: ID does not exist" Sep 16 01:04:31.782598 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.782527 2567 scope.go:117] "RemoveContainer" containerID="735d94fd1533eba001fec7b939d8de647c4cc95b278cb6b1bd77e39467ca4255" Sep 16 01:04:31.782837 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:04:31.782819 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735d94fd1533eba001fec7b939d8de647c4cc95b278cb6b1bd77e39467ca4255\": container with ID starting with 735d94fd1533eba001fec7b939d8de647c4cc95b278cb6b1bd77e39467ca4255 not found: ID does not exist" containerID="735d94fd1533eba001fec7b939d8de647c4cc95b278cb6b1bd77e39467ca4255" Sep 16 01:04:31.782928 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.782841 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735d94fd1533eba001fec7b939d8de647c4cc95b278cb6b1bd77e39467ca4255"} err="failed to get container status \"735d94fd1533eba001fec7b939d8de647c4cc95b278cb6b1bd77e39467ca4255\": rpc error: code = NotFound desc = could not find container \"735d94fd1533eba001fec7b939d8de647c4cc95b278cb6b1bd77e39467ca4255\": container with ID starting with 735d94fd1533eba001fec7b939d8de647c4cc95b278cb6b1bd77e39467ca4255 not found: ID does not exist" Sep 16 01:04:31.785426 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.785365 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l"] Sep 16 01:04:31.789189 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:31.789161 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-409cde-openshift-default-9678d6cb8-bnk2l"] Sep 16 01:04:32.119509 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:32.119474 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="177f8426-32c4-43a9-b689-7984a4b2cd0f" path="/var/lib/kubelet/pods/177f8426-32c4-43a9-b689-7984a4b2cd0f/volumes" Sep 16 01:04:35.266752 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:35.266718 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:35.449709 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:35.449673 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:41.100891 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:41.100841 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:41.168701 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:41.168668 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:41.831523 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:41.831486 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:42.845894 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:42.845857 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:43.611785 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:43.611731 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:44.295992 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:44.295953 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:51.968686 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:51.968654 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:52.004791 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:52.004752 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr"] Sep 16 01:04:52.005134 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:52.005104 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr" podUID="1de8ad32-3aca-4ec4-9b85-603c5e895461" containerName="payload-pre-processing" containerID="cri-o://c7404f629fe1ba15104e330b8d7fc4a5b701c9567a46f68b38465d466c39d1d1" gracePeriod=30 Sep 16 01:04:52.273911 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:52.273883 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr" Sep 16 01:04:52.367303 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:52.367261 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nh8t9\" (UniqueName: \"kubernetes.io/projected/1de8ad32-3aca-4ec4-9b85-603c5e895461-kube-api-access-nh8t9\") pod \"1de8ad32-3aca-4ec4-9b85-603c5e895461\" (UID: \"1de8ad32-3aca-4ec4-9b85-603c5e895461\") " Sep 16 01:04:52.367505 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:52.367367 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1de8ad32-3aca-4ec4-9b85-603c5e895461-plugins-config-volume\") pod \"1de8ad32-3aca-4ec4-9b85-603c5e895461\" (UID: \"1de8ad32-3aca-4ec4-9b85-603c5e895461\") " Sep 16 01:04:52.367736 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:52.367711 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1de8ad32-3aca-4ec4-9b85-603c5e895461-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1de8ad32-3aca-4ec4-9b85-603c5e895461" (UID: "1de8ad32-3aca-4ec4-9b85-603c5e895461"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:04:52.369723 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:52.369697 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1de8ad32-3aca-4ec4-9b85-603c5e895461-kube-api-access-nh8t9" (OuterVolumeSpecName: "kube-api-access-nh8t9") pod "1de8ad32-3aca-4ec4-9b85-603c5e895461" (UID: "1de8ad32-3aca-4ec4-9b85-603c5e895461"). InnerVolumeSpecName "kube-api-access-nh8t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:04:52.468389 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:52.468350 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1de8ad32-3aca-4ec4-9b85-603c5e895461-plugins-config-volume\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:04:52.468389 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:52.468381 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh8t9\" (UniqueName: \"kubernetes.io/projected/1de8ad32-3aca-4ec4-9b85-603c5e895461-kube-api-access-nh8t9\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:04:52.893462 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:52.893421 2567 generic.go:356] "Generic (PLEG): container finished" podID="1de8ad32-3aca-4ec4-9b85-603c5e895461" containerID="c7404f629fe1ba15104e330b8d7fc4a5b701c9567a46f68b38465d466c39d1d1" exitCode=0 Sep 16 01:04:52.893713 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:52.893490 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr" Sep 16 01:04:52.893713 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:52.893532 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr" event={"ID":"1de8ad32-3aca-4ec4-9b85-603c5e895461","Type":"ContainerDied","Data":"c7404f629fe1ba15104e330b8d7fc4a5b701c9567a46f68b38465d466c39d1d1"} Sep 16 01:04:52.893713 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:52.893602 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr" event={"ID":"1de8ad32-3aca-4ec4-9b85-603c5e895461","Type":"ContainerDied","Data":"83ea8f2c2d36fcd1375d0bdf266268a2df51d7744c97204fc1092d2f39cc5bab"} Sep 16 01:04:52.893713 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:52.893634 2567 scope.go:117] "RemoveContainer" containerID="c7404f629fe1ba15104e330b8d7fc4a5b701c9567a46f68b38465d466c39d1d1" Sep 16 01:04:52.903654 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:52.903629 2567 scope.go:117] "RemoveContainer" containerID="c7404f629fe1ba15104e330b8d7fc4a5b701c9567a46f68b38465d466c39d1d1" Sep 16 01:04:52.904083 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:04:52.904028 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7404f629fe1ba15104e330b8d7fc4a5b701c9567a46f68b38465d466c39d1d1\": container with ID starting with c7404f629fe1ba15104e330b8d7fc4a5b701c9567a46f68b38465d466c39d1d1 not found: ID does not exist" containerID="c7404f629fe1ba15104e330b8d7fc4a5b701c9567a46f68b38465d466c39d1d1" Sep 16 01:04:52.904141 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:52.904096 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7404f629fe1ba15104e330b8d7fc4a5b701c9567a46f68b38465d466c39d1d1"} err="failed to get container status \"c7404f629fe1ba15104e330b8d7fc4a5b701c9567a46f68b38465d466c39d1d1\": rpc error: code = NotFound desc = could not find container \"c7404f629fe1ba15104e330b8d7fc4a5b701c9567a46f68b38465d466c39d1d1\": container with ID starting with c7404f629fe1ba15104e330b8d7fc4a5b701c9567a46f68b38465d466c39d1d1 not found: ID does not exist" Sep 16 01:04:52.920274 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:52.920234 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr"] Sep 16 01:04:52.923942 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:52.923897 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0e8a7b-7bccf45d54-ncdqr"] Sep 16 01:04:53.824146 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:53.824093 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:54.119843 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:54.119757 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1de8ad32-3aca-4ec4-9b85-603c5e895461" path="/var/lib/kubelet/pods/1de8ad32-3aca-4ec4-9b85-603c5e895461/volumes" Sep 16 01:04:55.430977 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:55.430940 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:04:58.403066 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:58.403007 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:58.464805 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:58.464763 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:58.651312 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:58.651270 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws"] Sep 16 01:04:58.651664 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:58.651629 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" podUID="a678da49-afd0-4c92-9dd9-07f802b92e3e" containerName="istio-proxy" containerID="cri-o://b7160efc689eaf9e85517becdef22d1cb4a7b730e6e2baf563b6a8a780b4a16c" gracePeriod=30 Sep 16 01:04:58.672888 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:58.672767 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:04:59.313530 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:04:59.313471 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:00.226742 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:00.226699 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:01.100256 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:01.100225 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:03.914597 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.914518 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:05:03.959934 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.959903 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-workload-certs\") pod \"a678da49-afd0-4c92-9dd9-07f802b92e3e\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " Sep 16 01:05:03.959934 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.959939 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-data\") pod \"a678da49-afd0-4c92-9dd9-07f802b92e3e\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " Sep 16 01:05:03.960258 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.959988 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a678da49-afd0-4c92-9dd9-07f802b92e3e-istiod-ca-cert\") pod \"a678da49-afd0-4c92-9dd9-07f802b92e3e\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " Sep 16 01:05:03.960258 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.960014 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-podinfo\") pod \"a678da49-afd0-4c92-9dd9-07f802b92e3e\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " Sep 16 01:05:03.960258 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.960083 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-credential-socket\") pod \"a678da49-afd0-4c92-9dd9-07f802b92e3e\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " Sep 16 01:05:03.960258 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.960148 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-envoy\") pod \"a678da49-afd0-4c92-9dd9-07f802b92e3e\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " Sep 16 01:05:03.960258 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.960196 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-workload-socket\") pod \"a678da49-afd0-4c92-9dd9-07f802b92e3e\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " Sep 16 01:05:03.960258 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.960224 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-token\") pod \"a678da49-afd0-4c92-9dd9-07f802b92e3e\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " Sep 16 01:05:03.960258 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.960251 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v66hb\" (UniqueName: \"kubernetes.io/projected/a678da49-afd0-4c92-9dd9-07f802b92e3e-kube-api-access-v66hb\") pod \"a678da49-afd0-4c92-9dd9-07f802b92e3e\" (UID: \"a678da49-afd0-4c92-9dd9-07f802b92e3e\") " Sep 16 01:05:03.960600 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.960297 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a678da49-afd0-4c92-9dd9-07f802b92e3e" (UID: "a678da49-afd0-4c92-9dd9-07f802b92e3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:05:03.960600 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.960348 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-data" (OuterVolumeSpecName: "istio-data") pod "a678da49-afd0-4c92-9dd9-07f802b92e3e" (UID: "a678da49-afd0-4c92-9dd9-07f802b92e3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:05:03.960600 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.960583 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-workload-certs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:03.960768 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.960603 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-data\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:03.960768 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.960638 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a678da49-afd0-4c92-9dd9-07f802b92e3e" (UID: "a678da49-afd0-4c92-9dd9-07f802b92e3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:05:03.960768 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.960742 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a678da49-afd0-4c92-9dd9-07f802b92e3e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a678da49-afd0-4c92-9dd9-07f802b92e3e" (UID: "a678da49-afd0-4c92-9dd9-07f802b92e3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:05:03.961641 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.961022 2567 generic.go:356] "Generic (PLEG): container finished" podID="a678da49-afd0-4c92-9dd9-07f802b92e3e" containerID="b7160efc689eaf9e85517becdef22d1cb4a7b730e6e2baf563b6a8a780b4a16c" exitCode=0 Sep 16 01:05:03.961641 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.961190 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" event={"ID":"a678da49-afd0-4c92-9dd9-07f802b92e3e","Type":"ContainerDied","Data":"b7160efc689eaf9e85517becdef22d1cb4a7b730e6e2baf563b6a8a780b4a16c"} Sep 16 01:05:03.961641 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.961381 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" Sep 16 01:05:03.961641 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.961423 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws" event={"ID":"a678da49-afd0-4c92-9dd9-07f802b92e3e","Type":"ContainerDied","Data":"cbb5367a611e122ea2963b872ad7d1b67a151e7300ae7ee7bd09b23684d01933"} Sep 16 01:05:03.961641 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.961460 2567 scope.go:117] "RemoveContainer" containerID="b7160efc689eaf9e85517becdef22d1cb4a7b730e6e2baf563b6a8a780b4a16c" Sep 16 01:05:03.962029 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.961948 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a678da49-afd0-4c92-9dd9-07f802b92e3e" (UID: "a678da49-afd0-4c92-9dd9-07f802b92e3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:05:03.963217 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.963183 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a678da49-afd0-4c92-9dd9-07f802b92e3e" (UID: "a678da49-afd0-4c92-9dd9-07f802b92e3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:05:03.963393 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.963371 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a678da49-afd0-4c92-9dd9-07f802b92e3e" (UID: "a678da49-afd0-4c92-9dd9-07f802b92e3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 01:05:03.963486 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.963385 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-token" (OuterVolumeSpecName: "istio-token") pod "a678da49-afd0-4c92-9dd9-07f802b92e3e" (UID: "a678da49-afd0-4c92-9dd9-07f802b92e3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:05:03.963486 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.963429 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a678da49-afd0-4c92-9dd9-07f802b92e3e-kube-api-access-v66hb" (OuterVolumeSpecName: "kube-api-access-v66hb") pod "a678da49-afd0-4c92-9dd9-07f802b92e3e" (UID: "a678da49-afd0-4c92-9dd9-07f802b92e3e"). InnerVolumeSpecName "kube-api-access-v66hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:05:03.983524 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.983486 2567 scope.go:117] "RemoveContainer" containerID="7697b8768eb97e8c005cc77f358e609975f6efa0ddb70fb0ae2df25da08a21b6" Sep 16 01:05:03.993782 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.993757 2567 scope.go:117] "RemoveContainer" containerID="b7160efc689eaf9e85517becdef22d1cb4a7b730e6e2baf563b6a8a780b4a16c" Sep 16 01:05:03.994268 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:05:03.994244 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7160efc689eaf9e85517becdef22d1cb4a7b730e6e2baf563b6a8a780b4a16c\": container with ID starting with b7160efc689eaf9e85517becdef22d1cb4a7b730e6e2baf563b6a8a780b4a16c not found: ID does not exist" containerID="b7160efc689eaf9e85517becdef22d1cb4a7b730e6e2baf563b6a8a780b4a16c" Sep 16 01:05:03.994352 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.994280 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7160efc689eaf9e85517becdef22d1cb4a7b730e6e2baf563b6a8a780b4a16c"} err="failed to get container status \"b7160efc689eaf9e85517becdef22d1cb4a7b730e6e2baf563b6a8a780b4a16c\": rpc error: code = NotFound desc = could not find container \"b7160efc689eaf9e85517becdef22d1cb4a7b730e6e2baf563b6a8a780b4a16c\": container with ID starting with b7160efc689eaf9e85517becdef22d1cb4a7b730e6e2baf563b6a8a780b4a16c not found: ID does not exist" Sep 16 01:05:03.994352 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.994308 2567 scope.go:117] "RemoveContainer" containerID="7697b8768eb97e8c005cc77f358e609975f6efa0ddb70fb0ae2df25da08a21b6" Sep 16 01:05:03.994592 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:05:03.994579 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7697b8768eb97e8c005cc77f358e609975f6efa0ddb70fb0ae2df25da08a21b6\": container with ID starting with 7697b8768eb97e8c005cc77f358e609975f6efa0ddb70fb0ae2df25da08a21b6 not found: ID does not exist" containerID="7697b8768eb97e8c005cc77f358e609975f6efa0ddb70fb0ae2df25da08a21b6" Sep 16 01:05:03.994632 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:03.994597 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7697b8768eb97e8c005cc77f358e609975f6efa0ddb70fb0ae2df25da08a21b6"} err="failed to get container status \"7697b8768eb97e8c005cc77f358e609975f6efa0ddb70fb0ae2df25da08a21b6\": rpc error: code = NotFound desc = could not find container \"7697b8768eb97e8c005cc77f358e609975f6efa0ddb70fb0ae2df25da08a21b6\": container with ID starting with 7697b8768eb97e8c005cc77f358e609975f6efa0ddb70fb0ae2df25da08a21b6 not found: ID does not exist" Sep 16 01:05:04.061877 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:04.061838 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-credential-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:04.061877 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:04.061873 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-envoy\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:04.061877 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:04.061883 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a678da49-afd0-4c92-9dd9-07f802b92e3e-workload-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:04.062133 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:04.061893 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-token\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:04.062133 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:04.061902 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v66hb\" (UniqueName: \"kubernetes.io/projected/a678da49-afd0-4c92-9dd9-07f802b92e3e-kube-api-access-v66hb\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:04.062133 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:04.061911 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a678da49-afd0-4c92-9dd9-07f802b92e3e-istiod-ca-cert\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:04.062133 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:04.061919 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a678da49-afd0-4c92-9dd9-07f802b92e3e-istio-podinfo\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:04.280953 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:04.280783 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws"] Sep 16 01:05:04.284382 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:04.284346 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-0e8a7b-openshift-default-7f84ffc9b6-hj2ws"] Sep 16 01:05:06.119531 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:06.119495 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a678da49-afd0-4c92-9dd9-07f802b92e3e" path="/var/lib/kubelet/pods/a678da49-afd0-4c92-9dd9-07f802b92e3e/volumes" Sep 16 01:05:09.543417 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:09.543382 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:09.574585 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:09.574547 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v"] Sep 16 01:05:09.574863 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:09.574829 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v" podUID="08dd2134-017e-48bf-a2ac-96ee9984931e" containerName="payload-processing" containerID="cri-o://8fb868e44aadd38d9eb2beedd33ff7da8a5eb29170cf07cc64915e20e8307a0c" gracePeriod=30 Sep 16 01:05:09.611483 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:09.611449 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:09.711063 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:09.711013 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:09.799422 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:09.799328 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:09.832055 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:09.832006 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v" Sep 16 01:05:09.915467 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:09.915430 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08dd2134-017e-48bf-a2ac-96ee9984931e-plugins-config-volume\") pod \"08dd2134-017e-48bf-a2ac-96ee9984931e\" (UID: \"08dd2134-017e-48bf-a2ac-96ee9984931e\") " Sep 16 01:05:09.915467 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:09.915476 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsdlr\" (UniqueName: \"kubernetes.io/projected/08dd2134-017e-48bf-a2ac-96ee9984931e-kube-api-access-zsdlr\") pod \"08dd2134-017e-48bf-a2ac-96ee9984931e\" (UID: \"08dd2134-017e-48bf-a2ac-96ee9984931e\") " Sep 16 01:05:09.915950 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:09.915919 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08dd2134-017e-48bf-a2ac-96ee9984931e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "08dd2134-017e-48bf-a2ac-96ee9984931e" (UID: "08dd2134-017e-48bf-a2ac-96ee9984931e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:05:09.919018 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:09.918558 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08dd2134-017e-48bf-a2ac-96ee9984931e-kube-api-access-zsdlr" (OuterVolumeSpecName: "kube-api-access-zsdlr") pod "08dd2134-017e-48bf-a2ac-96ee9984931e" (UID: "08dd2134-017e-48bf-a2ac-96ee9984931e"). InnerVolumeSpecName "kube-api-access-zsdlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:05:09.999911 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:09.999874 2567 generic.go:356] "Generic (PLEG): container finished" podID="08dd2134-017e-48bf-a2ac-96ee9984931e" containerID="8fb868e44aadd38d9eb2beedd33ff7da8a5eb29170cf07cc64915e20e8307a0c" exitCode=0 Sep 16 01:05:10.000144 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:09.999922 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v" Sep 16 01:05:10.000144 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:09.999965 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v" event={"ID":"08dd2134-017e-48bf-a2ac-96ee9984931e","Type":"ContainerDied","Data":"8fb868e44aadd38d9eb2beedd33ff7da8a5eb29170cf07cc64915e20e8307a0c"} Sep 16 01:05:10.000144 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:10.000021 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v" event={"ID":"08dd2134-017e-48bf-a2ac-96ee9984931e","Type":"ContainerDied","Data":"4d70f89a7b270bebf3d1e4a2c365253129806829b1643184528811af6d9968af"} Sep 16 01:05:10.000144 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:10.000062 2567 scope.go:117] "RemoveContainer" containerID="8fb868e44aadd38d9eb2beedd33ff7da8a5eb29170cf07cc64915e20e8307a0c" Sep 16 01:05:10.010453 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:10.010432 2567 scope.go:117] "RemoveContainer" containerID="8fb868e44aadd38d9eb2beedd33ff7da8a5eb29170cf07cc64915e20e8307a0c" Sep 16 01:05:10.010834 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:05:10.010808 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb868e44aadd38d9eb2beedd33ff7da8a5eb29170cf07cc64915e20e8307a0c\": container with ID starting with 8fb868e44aadd38d9eb2beedd33ff7da8a5eb29170cf07cc64915e20e8307a0c not found: ID does not exist" containerID="8fb868e44aadd38d9eb2beedd33ff7da8a5eb29170cf07cc64915e20e8307a0c" Sep 16 01:05:10.010924 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:10.010842 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb868e44aadd38d9eb2beedd33ff7da8a5eb29170cf07cc64915e20e8307a0c"} err="failed to get container status \"8fb868e44aadd38d9eb2beedd33ff7da8a5eb29170cf07cc64915e20e8307a0c\": rpc error: code = NotFound desc = could not find container \"8fb868e44aadd38d9eb2beedd33ff7da8a5eb29170cf07cc64915e20e8307a0c\": container with ID starting with 8fb868e44aadd38d9eb2beedd33ff7da8a5eb29170cf07cc64915e20e8307a0c not found: ID does not exist" Sep 16 01:05:10.016443 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:10.016416 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08dd2134-017e-48bf-a2ac-96ee9984931e-plugins-config-volume\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:10.016443 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:10.016441 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsdlr\" (UniqueName: \"kubernetes.io/projected/08dd2134-017e-48bf-a2ac-96ee9984931e-kube-api-access-zsdlr\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:10.022522 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:10.022495 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v"] Sep 16 01:05:10.026154 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:10.026122 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-108960-6b6b657989-xgr9v"] Sep 16 01:05:10.108016 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:10.107980 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:10.125945 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:10.125912 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08dd2134-017e-48bf-a2ac-96ee9984931e" path="/var/lib/kubelet/pods/08dd2134-017e-48bf-a2ac-96ee9984931e/volumes" Sep 16 01:05:10.327803 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:10.327760 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:11.934152 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:11.934108 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:05:15.903692 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:15.903659 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:16.003386 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.003340 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:16.181798 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.181672 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:16.521375 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521281 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2fa86d-2vtnw"] Sep 16 01:05:16.521658 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521645 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="124e2e72-5d99-4745-92a0-a6624da3ea07" containerName="istio-proxy" Sep 16 01:05:16.521709 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521661 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="124e2e72-5d99-4745-92a0-a6624da3ea07" containerName="istio-proxy" Sep 16 01:05:16.521709 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521673 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aba147f-9352-4b2b-9e23-7eac729a8fd8" containerName="istio-proxy" Sep 16 01:05:16.521709 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521678 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aba147f-9352-4b2b-9e23-7eac729a8fd8" containerName="istio-proxy" Sep 16 01:05:16.521709 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521687 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfcd3554-65e7-45c1-94f5-df99995fead2" containerName="main" Sep 16 01:05:16.521709 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521693 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfcd3554-65e7-45c1-94f5-df99995fead2" containerName="main" Sep 16 01:05:16.521709 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521702 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="177f8426-32c4-43a9-b689-7984a4b2cd0f" containerName="istio-proxy" Sep 16 01:05:16.521709 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521708 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="177f8426-32c4-43a9-b689-7984a4b2cd0f" containerName="istio-proxy" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521718 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16067f7e-16b1-42e4-8bea-54a88f0ffbd9" containerName="payload-processing" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521724 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="16067f7e-16b1-42e4-8bea-54a88f0ffbd9" containerName="payload-processing" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521734 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21cb4d08-16e2-4d5a-9f9e-59c394282a26" containerName="payload-processing" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521740 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="21cb4d08-16e2-4d5a-9f9e-59c394282a26" containerName="payload-processing" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521747 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b789106-a743-457f-9272-019778117af0" containerName="istio-proxy" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521752 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b789106-a743-457f-9272-019778117af0" containerName="istio-proxy" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521758 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="124e2e72-5d99-4745-92a0-a6624da3ea07" containerName="istio-proxy" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521763 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="124e2e72-5d99-4745-92a0-a6624da3ea07" containerName="istio-proxy" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521770 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de8ad32-3aca-4ec4-9b85-603c5e895461" containerName="payload-pre-processing" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521775 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de8ad32-3aca-4ec4-9b85-603c5e895461" containerName="payload-pre-processing" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521785 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a678da49-afd0-4c92-9dd9-07f802b92e3e" containerName="istio-proxy" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521790 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a678da49-afd0-4c92-9dd9-07f802b92e3e" containerName="istio-proxy" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521797 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08dd2134-017e-48bf-a2ac-96ee9984931e" containerName="payload-processing" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521802 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="08dd2134-017e-48bf-a2ac-96ee9984931e" containerName="payload-processing" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521808 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a678da49-afd0-4c92-9dd9-07f802b92e3e" containerName="istio-proxy" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521813 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a678da49-afd0-4c92-9dd9-07f802b92e3e" containerName="istio-proxy" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521864 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aba147f-9352-4b2b-9e23-7eac729a8fd8" containerName="istio-proxy" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521871 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a678da49-afd0-4c92-9dd9-07f802b92e3e" containerName="istio-proxy" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521879 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a678da49-afd0-4c92-9dd9-07f802b92e3e" containerName="istio-proxy" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521886 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="124e2e72-5d99-4745-92a0-a6624da3ea07" containerName="istio-proxy" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521892 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b789106-a743-457f-9272-019778117af0" containerName="istio-proxy" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521897 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="08dd2134-017e-48bf-a2ac-96ee9984931e" containerName="payload-processing" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521903 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="21cb4d08-16e2-4d5a-9f9e-59c394282a26" containerName="payload-processing" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521908 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfcd3554-65e7-45c1-94f5-df99995fead2" containerName="main" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521913 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="16067f7e-16b1-42e4-8bea-54a88f0ffbd9" containerName="payload-processing" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521919 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de8ad32-3aca-4ec4-9b85-603c5e895461" containerName="payload-pre-processing" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521925 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="177f8426-32c4-43a9-b689-7984a4b2cd0f" containerName="istio-proxy" Sep 16 01:05:16.521920 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.521930 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="177f8426-32c4-43a9-b689-7984a4b2cd0f" containerName="istio-proxy" Sep 16 01:05:16.524692 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.524672 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2fa86d-2vtnw" Sep 16 01:05:16.527393 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.527368 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 01:05:16.527534 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.527504 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-npltg\"" Sep 16 01:05:16.532462 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.532420 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2fa86d-2vtnw"] Sep 16 01:05:16.581962 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.581929 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9be240-75f2-46b2-8f31-12c4fa74ea53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2fa86d-2vtnw\" (UID: \"0a9be240-75f2-46b2-8f31-12c4fa74ea53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2fa86d-2vtnw" Sep 16 01:05:16.683243 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.683206 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9be240-75f2-46b2-8f31-12c4fa74ea53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2fa86d-2vtnw\" (UID: \"0a9be240-75f2-46b2-8f31-12c4fa74ea53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2fa86d-2vtnw" Sep 16 01:05:16.683907 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.683874 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9be240-75f2-46b2-8f31-12c4fa74ea53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2fa86d-2vtnw\" (UID: \"0a9be240-75f2-46b2-8f31-12c4fa74ea53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2fa86d-2vtnw" Sep 16 01:05:16.837711 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:16.837669 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2fa86d-2vtnw" Sep 16 01:05:17.115755 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:17.115653 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2fa86d-2vtnw"] Sep 16 01:05:17.118790 ip-10-0-131-194 kubenswrapper[2567]: W0916 01:05:17.118758 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9be240_75f2_46b2_8f31_12c4fa74ea53.slice/crio-8c870fb74188b2e88e6a9cfb071dd7938f27f4a3a18c6b8b940d1a1d45816747 WatchSource:0}: Error finding container 8c870fb74188b2e88e6a9cfb071dd7938f27f4a3a18c6b8b940d1a1d45816747: Status 404 returned error can't find the container with id 8c870fb74188b2e88e6a9cfb071dd7938f27f4a3a18c6b8b940d1a1d45816747 Sep 16 01:05:18.051298 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:18.051261 2567 generic.go:356] "Generic (PLEG): container finished" podID="0a9be240-75f2-46b2-8f31-12c4fa74ea53" containerID="5114413156a514653bd9badcef76a54626cbf6effdf02e2b9190efa4e87337b8" exitCode=0 Sep 16 01:05:18.051497 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:18.051306 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2fa86d-2vtnw" event={"ID":"0a9be240-75f2-46b2-8f31-12c4fa74ea53","Type":"ContainerDied","Data":"5114413156a514653bd9badcef76a54626cbf6effdf02e2b9190efa4e87337b8"} Sep 16 01:05:18.051497 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:18.051367 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2fa86d-2vtnw" event={"ID":"0a9be240-75f2-46b2-8f31-12c4fa74ea53","Type":"ContainerStarted","Data":"8c870fb74188b2e88e6a9cfb071dd7938f27f4a3a18c6b8b940d1a1d45816747"} Sep 16 01:05:19.192275 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:19.192248 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2fa86d-2vtnw" Sep 16 01:05:19.310277 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:19.310175 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9be240-75f2-46b2-8f31-12c4fa74ea53-maas-api-service-ca\") pod \"0a9be240-75f2-46b2-8f31-12c4fa74ea53\" (UID: \"0a9be240-75f2-46b2-8f31-12c4fa74ea53\") " Sep 16 01:05:19.310568 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:19.310541 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9be240-75f2-46b2-8f31-12c4fa74ea53-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a9be240-75f2-46b2-8f31-12c4fa74ea53" (UID: "0a9be240-75f2-46b2-8f31-12c4fa74ea53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:05:19.410877 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:19.410843 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9be240-75f2-46b2-8f31-12c4fa74ea53-maas-api-service-ca\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:20.065736 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:20.065700 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2fa86d-2vtnw" Sep 16 01:05:20.065736 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:20.065723 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2fa86d-2vtnw" event={"ID":"0a9be240-75f2-46b2-8f31-12c4fa74ea53","Type":"ContainerDied","Data":"8c870fb74188b2e88e6a9cfb071dd7938f27f4a3a18c6b8b940d1a1d45816747"} Sep 16 01:05:20.065968 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:20.065774 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c870fb74188b2e88e6a9cfb071dd7938f27f4a3a18c6b8b940d1a1d45816747" Sep 16 01:05:21.856711 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:21.856670 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:26.983719 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:26.983686 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:26.988142 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:26.988094 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2fa86d-29825340-d284r"] Sep 16 01:05:26.990519 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:26.990487 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2fa86d-29825340-d284r"] Sep 16 01:05:27.029231 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:27.020514 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22"] Sep 16 01:05:27.029231 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:27.020888 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22" podUID="15d05ebf-2444-4d61-9e69-94afc252b036" containerName="payload-pre-processing" containerID="cri-o://194f1b24443d78ee54ea3aa60782fb101f454e6a98b4f37ed498783d935af804" gracePeriod=30 Sep 16 01:05:27.083315 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:27.083277 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:27.169611 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:27.169569 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2fa86d-2vtnw"] Sep 16 01:05:27.170996 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:27.170963 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:27.172095 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:27.172067 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2fa86d-2vtnw"] Sep 16 01:05:27.229497 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:27.229458 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:27.310121 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:27.310091 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22" Sep 16 01:05:27.392728 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:27.392688 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d05ebf-2444-4d61-9e69-94afc252b036-plugins-config-volume\") pod \"15d05ebf-2444-4d61-9e69-94afc252b036\" (UID: \"15d05ebf-2444-4d61-9e69-94afc252b036\") " Sep 16 01:05:27.392933 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:27.392756 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xntmx\" (UniqueName: \"kubernetes.io/projected/15d05ebf-2444-4d61-9e69-94afc252b036-kube-api-access-xntmx\") pod \"15d05ebf-2444-4d61-9e69-94afc252b036\" (UID: \"15d05ebf-2444-4d61-9e69-94afc252b036\") " Sep 16 01:05:27.393133 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:27.393110 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15d05ebf-2444-4d61-9e69-94afc252b036-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "15d05ebf-2444-4d61-9e69-94afc252b036" (UID: "15d05ebf-2444-4d61-9e69-94afc252b036"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:05:27.395130 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:27.395099 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d05ebf-2444-4d61-9e69-94afc252b036-kube-api-access-xntmx" (OuterVolumeSpecName: "kube-api-access-xntmx") pod "15d05ebf-2444-4d61-9e69-94afc252b036" (UID: "15d05ebf-2444-4d61-9e69-94afc252b036"). InnerVolumeSpecName "kube-api-access-xntmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:05:27.493694 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:27.493650 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d05ebf-2444-4d61-9e69-94afc252b036-plugins-config-volume\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:27.493694 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:27.493687 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xntmx\" (UniqueName: \"kubernetes.io/projected/15d05ebf-2444-4d61-9e69-94afc252b036-kube-api-access-xntmx\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:27.589873 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:27.589841 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:28.112691 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:28.112641 2567 generic.go:356] "Generic (PLEG): container finished" podID="15d05ebf-2444-4d61-9e69-94afc252b036" containerID="194f1b24443d78ee54ea3aa60782fb101f454e6a98b4f37ed498783d935af804" exitCode=0 Sep 16 01:05:28.113082 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:28.112759 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22" Sep 16 01:05:28.119089 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:28.119061 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a9be240-75f2-46b2-8f31-12c4fa74ea53" path="/var/lib/kubelet/pods/0a9be240-75f2-46b2-8f31-12c4fa74ea53/volumes" Sep 16 01:05:28.119380 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:28.119368 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ba1d72e-99f5-4d36-994a-d806e28cd1bb" path="/var/lib/kubelet/pods/8ba1d72e-99f5-4d36-994a-d806e28cd1bb/volumes" Sep 16 01:05:28.119685 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:28.119664 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22" event={"ID":"15d05ebf-2444-4d61-9e69-94afc252b036","Type":"ContainerDied","Data":"194f1b24443d78ee54ea3aa60782fb101f454e6a98b4f37ed498783d935af804"} Sep 16 01:05:28.119806 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:28.119709 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22" event={"ID":"15d05ebf-2444-4d61-9e69-94afc252b036","Type":"ContainerDied","Data":"2eb5752505504e86e1593a67af2ff56e47b425ae4e28d91c00f2b803069df81d"} Sep 16 01:05:28.119806 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:28.119733 2567 scope.go:117] "RemoveContainer" containerID="194f1b24443d78ee54ea3aa60782fb101f454e6a98b4f37ed498783d935af804" Sep 16 01:05:28.129271 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:28.129248 2567 scope.go:117] "RemoveContainer" containerID="194f1b24443d78ee54ea3aa60782fb101f454e6a98b4f37ed498783d935af804" Sep 16 01:05:28.129617 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:05:28.129595 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"194f1b24443d78ee54ea3aa60782fb101f454e6a98b4f37ed498783d935af804\": container with ID starting with 194f1b24443d78ee54ea3aa60782fb101f454e6a98b4f37ed498783d935af804 not found: ID does not exist" containerID="194f1b24443d78ee54ea3aa60782fb101f454e6a98b4f37ed498783d935af804" Sep 16 01:05:28.129709 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:28.129630 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"194f1b24443d78ee54ea3aa60782fb101f454e6a98b4f37ed498783d935af804"} err="failed to get container status \"194f1b24443d78ee54ea3aa60782fb101f454e6a98b4f37ed498783d935af804\": rpc error: code = NotFound desc = could not find container \"194f1b24443d78ee54ea3aa60782fb101f454e6a98b4f37ed498783d935af804\": container with ID starting with 194f1b24443d78ee54ea3aa60782fb101f454e6a98b4f37ed498783d935af804 not found: ID does not exist" Sep 16 01:05:28.141219 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:28.141182 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22"] Sep 16 01:05:28.145377 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:28.145347 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2fa86d-6755c7c75d-tvd22"] Sep 16 01:05:29.637048 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:29.637008 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:05:30.121551 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:30.121514 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15d05ebf-2444-4d61-9e69-94afc252b036" path="/var/lib/kubelet/pods/15d05ebf-2444-4d61-9e69-94afc252b036/volumes" Sep 16 01:05:34.014507 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:34.014395 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:34.115868 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:34.115833 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:34.295362 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:34.295265 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8"] Sep 16 01:05:34.295546 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:34.295489 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" podUID="0f89837a-e1cf-4d80-a25c-145602b37ca7" containerName="istio-proxy" containerID="cri-o://7d9e537a3759b4a43855cecfdd4d64d594cb54e233b7d2e318fcaa1338553637" gracePeriod=30 Sep 16 01:05:34.320789 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:34.320738 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:05:39.127708 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.127666 2567 patch_prober.go:28] interesting pod/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 01:05:39.128120 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.127733 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" podUID="0f89837a-e1cf-4d80-a25c-145602b37ca7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 01:05:39.244156 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.244110 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:05:39.545794 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.545758 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 01:05:39.613824 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.613783 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-podinfo\") pod \"0f89837a-e1cf-4d80-a25c-145602b37ca7\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " Sep 16 01:05:39.613824 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.613837 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-workload-certs\") pod \"0f89837a-e1cf-4d80-a25c-145602b37ca7\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " Sep 16 01:05:39.614144 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.613872 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-data\") pod \"0f89837a-e1cf-4d80-a25c-145602b37ca7\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " Sep 16 01:05:39.614144 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.613895 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-envoy\") pod \"0f89837a-e1cf-4d80-a25c-145602b37ca7\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " Sep 16 01:05:39.614144 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.613912 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rspj\" (UniqueName: \"kubernetes.io/projected/0f89837a-e1cf-4d80-a25c-145602b37ca7-kube-api-access-5rspj\") pod \"0f89837a-e1cf-4d80-a25c-145602b37ca7\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " Sep 16 01:05:39.614144 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.613936 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f89837a-e1cf-4d80-a25c-145602b37ca7-istiod-ca-cert\") pod \"0f89837a-e1cf-4d80-a25c-145602b37ca7\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " Sep 16 01:05:39.614144 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.613961 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-token\") pod \"0f89837a-e1cf-4d80-a25c-145602b37ca7\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " Sep 16 01:05:39.614144 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.614007 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-credential-socket\") pod \"0f89837a-e1cf-4d80-a25c-145602b37ca7\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " Sep 16 01:05:39.614144 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.614117 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-workload-socket\") pod \"0f89837a-e1cf-4d80-a25c-145602b37ca7\" (UID: \"0f89837a-e1cf-4d80-a25c-145602b37ca7\") " Sep 16 01:05:39.614514 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.614260 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-data" (OuterVolumeSpecName: "istio-data") pod "0f89837a-e1cf-4d80-a25c-145602b37ca7" (UID: "0f89837a-e1cf-4d80-a25c-145602b37ca7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:05:39.614514 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.614384 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0f89837a-e1cf-4d80-a25c-145602b37ca7" (UID: "0f89837a-e1cf-4d80-a25c-145602b37ca7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:05:39.614514 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.614412 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-data\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:39.614514 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.614399 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0f89837a-e1cf-4d80-a25c-145602b37ca7" (UID: "0f89837a-e1cf-4d80-a25c-145602b37ca7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:05:39.614888 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.614622 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0f89837a-e1cf-4d80-a25c-145602b37ca7" (UID: "0f89837a-e1cf-4d80-a25c-145602b37ca7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:05:39.614888 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.614652 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f89837a-e1cf-4d80-a25c-145602b37ca7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0f89837a-e1cf-4d80-a25c-145602b37ca7" (UID: "0f89837a-e1cf-4d80-a25c-145602b37ca7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:05:39.616396 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.616371 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0f89837a-e1cf-4d80-a25c-145602b37ca7" (UID: "0f89837a-e1cf-4d80-a25c-145602b37ca7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:05:39.616532 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.616508 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0f89837a-e1cf-4d80-a25c-145602b37ca7" (UID: "0f89837a-e1cf-4d80-a25c-145602b37ca7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 01:05:39.616583 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.616523 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-token" (OuterVolumeSpecName: "istio-token") pod "0f89837a-e1cf-4d80-a25c-145602b37ca7" (UID: "0f89837a-e1cf-4d80-a25c-145602b37ca7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:05:39.616623 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.616591 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f89837a-e1cf-4d80-a25c-145602b37ca7-kube-api-access-5rspj" (OuterVolumeSpecName: "kube-api-access-5rspj") pod "0f89837a-e1cf-4d80-a25c-145602b37ca7" (UID: "0f89837a-e1cf-4d80-a25c-145602b37ca7"). InnerVolumeSpecName "kube-api-access-5rspj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:05:39.715190 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.715088 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-token\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:39.715190 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.715122 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-credential-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:39.715190 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.715133 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-workload-socket\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:39.715190 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.715144 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-podinfo\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:39.715190 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.715152 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-workload-certs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:39.715190 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.715161 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f89837a-e1cf-4d80-a25c-145602b37ca7-istio-envoy\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:39.715190 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.715168 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rspj\" (UniqueName: \"kubernetes.io/projected/0f89837a-e1cf-4d80-a25c-145602b37ca7-kube-api-access-5rspj\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:39.715190 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:39.715177 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f89837a-e1cf-4d80-a25c-145602b37ca7-istiod-ca-cert\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:05:40.188718 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:40.188682 2567 generic.go:356] "Generic (PLEG): container finished" podID="0f89837a-e1cf-4d80-a25c-145602b37ca7" containerID="7d9e537a3759b4a43855cecfdd4d64d594cb54e233b7d2e318fcaa1338553637" exitCode=0 Sep 16 01:05:40.189227 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:40.188762 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" event={"ID":"0f89837a-e1cf-4d80-a25c-145602b37ca7","Type":"ContainerDied","Data":"7d9e537a3759b4a43855cecfdd4d64d594cb54e233b7d2e318fcaa1338553637"} Sep 16 01:05:40.189227 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:40.188793 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" Sep 16 01:05:40.189227 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:40.188801 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8" event={"ID":"0f89837a-e1cf-4d80-a25c-145602b37ca7","Type":"ContainerDied","Data":"4d51d99f9b12cb19c03fe27f788e3fc4b8af0b141a5a94de2657a1f2edaac0dd"} Sep 16 01:05:40.189227 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:40.188830 2567 scope.go:117] "RemoveContainer" containerID="7d9e537a3759b4a43855cecfdd4d64d594cb54e233b7d2e318fcaa1338553637" Sep 16 01:05:40.198575 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:40.198547 2567 scope.go:117] "RemoveContainer" containerID="18e6ebca4deef2ef900e844b5909b463b99da48dcd0879cf816874aedd447665" Sep 16 01:05:40.209115 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:40.209090 2567 scope.go:117] "RemoveContainer" containerID="7d9e537a3759b4a43855cecfdd4d64d594cb54e233b7d2e318fcaa1338553637" Sep 16 01:05:40.209535 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:05:40.209514 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d9e537a3759b4a43855cecfdd4d64d594cb54e233b7d2e318fcaa1338553637\": container with ID starting with 7d9e537a3759b4a43855cecfdd4d64d594cb54e233b7d2e318fcaa1338553637 not found: ID does not exist" containerID="7d9e537a3759b4a43855cecfdd4d64d594cb54e233b7d2e318fcaa1338553637" Sep 16 01:05:40.209597 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:40.209548 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d9e537a3759b4a43855cecfdd4d64d594cb54e233b7d2e318fcaa1338553637"} err="failed to get container status \"7d9e537a3759b4a43855cecfdd4d64d594cb54e233b7d2e318fcaa1338553637\": rpc error: code = NotFound desc = could not find container \"7d9e537a3759b4a43855cecfdd4d64d594cb54e233b7d2e318fcaa1338553637\": container with ID starting with 7d9e537a3759b4a43855cecfdd4d64d594cb54e233b7d2e318fcaa1338553637 not found: ID does not exist" Sep 16 01:05:40.209597 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:40.209573 2567 scope.go:117] "RemoveContainer" containerID="18e6ebca4deef2ef900e844b5909b463b99da48dcd0879cf816874aedd447665" Sep 16 01:05:40.209967 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:05:40.209937 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e6ebca4deef2ef900e844b5909b463b99da48dcd0879cf816874aedd447665\": container with ID starting with 18e6ebca4deef2ef900e844b5909b463b99da48dcd0879cf816874aedd447665 not found: ID does not exist" containerID="18e6ebca4deef2ef900e844b5909b463b99da48dcd0879cf816874aedd447665" Sep 16 01:05:40.210095 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:40.209980 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e6ebca4deef2ef900e844b5909b463b99da48dcd0879cf816874aedd447665"} err="failed to get container status \"18e6ebca4deef2ef900e844b5909b463b99da48dcd0879cf816874aedd447665\": rpc error: code = NotFound desc = could not find container \"18e6ebca4deef2ef900e844b5909b463b99da48dcd0879cf816874aedd447665\": container with ID starting with 18e6ebca4deef2ef900e844b5909b463b99da48dcd0879cf816874aedd447665 not found: ID does not exist" Sep 16 01:05:40.211388 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:40.211361 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8"] Sep 16 01:05:40.214209 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:40.214183 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2fa86d-openshift-default-6d899d88c4-cnnf8"] Sep 16 01:05:42.118453 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:42.118414 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f89837a-e1cf-4d80-a25c-145602b37ca7" path="/var/lib/kubelet/pods/0f89837a-e1cf-4d80-a25c-145602b37ca7/volumes" Sep 16 01:05:54.088583 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:05:54.088540 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:06:10.573763 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:06:10.573718 2567 scope.go:117] "RemoveContainer" containerID="8d984acaaf88b0bc439c34ef0a27b7d4871d42546c683a44cff4ec2703634586" Sep 16 01:06:10.582895 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:06:10.582867 2567 scope.go:117] "RemoveContainer" containerID="c2022217ce2ddef537c3ae3175197674efe4df744cfe4a6b2ecc4c79bd642cd7" Sep 16 01:06:10.592116 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:06:10.592088 2567 scope.go:117] "RemoveContainer" containerID="b3c6fa549c5a0186beae1d7bfe0336be7c0ac37facfff110aef868d8d28e93d5" Sep 16 01:06:29.291801 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:06:29.291759 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:06:29.344639 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:06:29.344606 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:06:57.808822 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:06:57.808787 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:06:57.951229 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:06:57.951181 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:06:59.410345 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:06:59.410312 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:07:23.645006 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:07:23.644903 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:07:31.665716 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:07:31.665670 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:07:39.364588 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:07:39.364553 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:07:47.745633 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:07:47.745583 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:08:01.943991 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:08:01.943959 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:08:10.442001 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:08:10.441969 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:09:38.730390 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:09:38.730351 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:09:43.214745 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:09:43.214714 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:09:43.264944 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:09:43.264918 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:09:47.047968 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:09:47.047931 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:09:53.140666 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:09:53.140629 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:10:00.699953 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.699919 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q"] Sep 16 01:10:00.700478 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.700459 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15d05ebf-2444-4d61-9e69-94afc252b036" containerName="payload-pre-processing" Sep 16 01:10:00.700558 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.700482 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="15d05ebf-2444-4d61-9e69-94afc252b036" containerName="payload-pre-processing" Sep 16 01:10:00.700558 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.700503 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aba147f-9352-4b2b-9e23-7eac729a8fd8" containerName="istio-proxy" Sep 16 01:10:00.700558 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.700512 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aba147f-9352-4b2b-9e23-7eac729a8fd8" containerName="istio-proxy" Sep 16 01:10:00.700558 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.700526 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="177f8426-32c4-43a9-b689-7984a4b2cd0f" containerName="istio-proxy" Sep 16 01:10:00.700558 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.700535 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="177f8426-32c4-43a9-b689-7984a4b2cd0f" containerName="istio-proxy" Sep 16 01:10:00.700558 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.700546 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a9be240-75f2-46b2-8f31-12c4fa74ea53" containerName="revoke-keys" Sep 16 01:10:00.700558 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.700554 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a9be240-75f2-46b2-8f31-12c4fa74ea53" containerName="revoke-keys" Sep 16 01:10:00.700900 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.700568 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f89837a-e1cf-4d80-a25c-145602b37ca7" containerName="istio-proxy" Sep 16 01:10:00.700900 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.700576 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f89837a-e1cf-4d80-a25c-145602b37ca7" containerName="istio-proxy" Sep 16 01:10:00.700900 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.700589 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f89837a-e1cf-4d80-a25c-145602b37ca7" containerName="istio-proxy" Sep 16 01:10:00.700900 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.700597 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f89837a-e1cf-4d80-a25c-145602b37ca7" containerName="istio-proxy" Sep 16 01:10:00.700900 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.700605 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b789106-a743-457f-9272-019778117af0" containerName="istio-proxy" Sep 16 01:10:00.700900 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.700613 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b789106-a743-457f-9272-019778117af0" containerName="istio-proxy" Sep 16 01:10:00.700900 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.700695 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aba147f-9352-4b2b-9e23-7eac729a8fd8" containerName="istio-proxy" Sep 16 01:10:00.700900 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.700706 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f89837a-e1cf-4d80-a25c-145602b37ca7" containerName="istio-proxy" Sep 16 01:10:00.700900 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.700714 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a9be240-75f2-46b2-8f31-12c4fa74ea53" containerName="revoke-keys" Sep 16 01:10:00.700900 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.700725 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="124e2e72-5d99-4745-92a0-a6624da3ea07" containerName="istio-proxy" Sep 16 01:10:00.700900 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.700734 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b789106-a743-457f-9272-019778117af0" containerName="istio-proxy" Sep 16 01:10:00.700900 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.700744 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="15d05ebf-2444-4d61-9e69-94afc252b036" containerName="payload-pre-processing" Sep 16 01:10:00.704007 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.703986 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:00.706928 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.706905 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-0a42620d-kserve-self-signed-certs\"" Sep 16 01:10:00.714693 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.714670 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q"] Sep 16 01:10:00.718031 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.718007 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:10:00.755720 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.755694 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-dshm\") pod \"e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:00.755828 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.755732 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-tmp-dir\") pod \"e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:00.755828 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.755763 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-home\") pod \"e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:00.755902 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.755823 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjnhs\" (UniqueName: \"kubernetes.io/projected/31db999d-78c1-4635-9d71-a17f4a4a4782-kube-api-access-xjnhs\") pod \"e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:00.755902 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.755857 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-model-cache\") pod \"e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:00.755902 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.755895 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31db999d-78c1-4635-9d71-a17f4a4a4782-tls-certs\") pod \"e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:00.856838 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.856805 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-tmp-dir\") pod \"e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:00.857018 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.856861 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-home\") pod \"e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:00.857018 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.856885 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjnhs\" (UniqueName: \"kubernetes.io/projected/31db999d-78c1-4635-9d71-a17f4a4a4782-kube-api-access-xjnhs\") pod \"e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:00.857018 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.856902 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-model-cache\") pod \"e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:00.857018 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.856944 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31db999d-78c1-4635-9d71-a17f4a4a4782-tls-certs\") pod \"e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:00.857018 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.856994 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-dshm\") pod \"e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:00.857353 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.857299 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-tmp-dir\") pod \"e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:00.857353 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.857328 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-home\") pod \"e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:00.857428 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.857355 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-model-cache\") pod \"e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:00.859483 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.859455 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-dshm\") pod \"e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:00.859627 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.859609 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31db999d-78c1-4635-9d71-a17f4a4a4782-tls-certs\") pod \"e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:00.864752 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:00.864735 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjnhs\" (UniqueName: \"kubernetes.io/projected/31db999d-78c1-4635-9d71-a17f4a4a4782-kube-api-access-xjnhs\") pod \"e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:01.014850 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:01.014770 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:01.288201 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:01.288085 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q"] Sep 16 01:10:01.290894 ip-10-0-131-194 kubenswrapper[2567]: W0916 01:10:01.290868 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31db999d_78c1_4635_9d71_a17f4a4a4782.slice/crio-3400985ef6f14a6876f75e48c1201dc2a41e4e205f332dde23c4380c0194a70d WatchSource:0}: Error finding container 3400985ef6f14a6876f75e48c1201dc2a41e4e205f332dde23c4380c0194a70d: Status 404 returned error can't find the container with id 3400985ef6f14a6876f75e48c1201dc2a41e4e205f332dde23c4380c0194a70d Sep 16 01:10:01.581629 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:01.581592 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" event={"ID":"31db999d-78c1-4635-9d71-a17f4a4a4782","Type":"ContainerStarted","Data":"30d17d814f5cce29ee1df9781ec9bad86cac1f14d18e8ef0724c79f5c08955a0"} Sep 16 01:10:01.581791 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:01.581650 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" event={"ID":"31db999d-78c1-4635-9d71-a17f4a4a4782","Type":"ContainerStarted","Data":"3400985ef6f14a6876f75e48c1201dc2a41e4e205f332dde23c4380c0194a70d"} Sep 16 01:10:01.599294 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:01.599246 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" podStartSLOduration=1.599232079 podStartE2EDuration="1.599232079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:10:01.597924994 +0000 UTC m=+2151.960148546" watchObservedRunningTime="2026-09-16 01:10:01.599232079 +0000 UTC m=+2151.961455610" Sep 16 01:10:02.360964 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:02.360929 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:10:02.427715 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:02.427696 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:10:03.340063 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:03.340016 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:10:10.337008 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:10.336978 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:10:11.015223 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:11.015188 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:11.015467 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:11.015438 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:11.027799 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:11.027775 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:11.644099 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:11.644070 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:10:12.341995 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:12.341954 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:10:19.575824 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:19.575793 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:10:38.157549 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:38.157513 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:10:46.278012 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:46.277978 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q"] Sep 16 01:10:46.304746 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:46.304713 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:10:47.765284 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:47.765249 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:10:48.641872 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:10:48.641838 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:11:01.297812 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.297772 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" podUID="31db999d-78c1-4635-9d71-a17f4a4a4782" containerName="main" containerID="cri-o://30d17d814f5cce29ee1df9781ec9bad86cac1f14d18e8ef0724c79f5c08955a0" gracePeriod=45 Sep 16 01:11:01.538621 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.538598 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:11:01.635704 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.635664 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:11:01.659503 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.659473 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31db999d-78c1-4635-9d71-a17f4a4a4782-tls-certs\") pod \"31db999d-78c1-4635-9d71-a17f4a4a4782\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " Sep 16 01:11:01.659649 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.659509 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-dshm\") pod \"31db999d-78c1-4635-9d71-a17f4a4a4782\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " Sep 16 01:11:01.659649 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.659536 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-home\") pod \"31db999d-78c1-4635-9d71-a17f4a4a4782\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " Sep 16 01:11:01.659649 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.659569 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-model-cache\") pod \"31db999d-78c1-4635-9d71-a17f4a4a4782\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " Sep 16 01:11:01.659821 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.659665 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjnhs\" (UniqueName: \"kubernetes.io/projected/31db999d-78c1-4635-9d71-a17f4a4a4782-kube-api-access-xjnhs\") pod \"31db999d-78c1-4635-9d71-a17f4a4a4782\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " Sep 16 01:11:01.659821 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.659758 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-tmp-dir\") pod \"31db999d-78c1-4635-9d71-a17f4a4a4782\" (UID: \"31db999d-78c1-4635-9d71-a17f4a4a4782\") " Sep 16 01:11:01.659936 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.659876 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-model-cache" (OuterVolumeSpecName: "model-cache") pod "31db999d-78c1-4635-9d71-a17f4a4a4782" (UID: "31db999d-78c1-4635-9d71-a17f4a4a4782"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:11:01.659936 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.659891 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-home" (OuterVolumeSpecName: "home") pod "31db999d-78c1-4635-9d71-a17f4a4a4782" (UID: "31db999d-78c1-4635-9d71-a17f4a4a4782"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:11:01.660078 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.660020 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31db999d-78c1-4635-9d71-a17f4a4a4782" (UID: "31db999d-78c1-4635-9d71-a17f4a4a4782"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:11:01.660138 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.660075 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-home\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:11:01.660138 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.660093 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-model-cache\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:11:01.661961 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.661937 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31db999d-78c1-4635-9d71-a17f4a4a4782-kube-api-access-xjnhs" (OuterVolumeSpecName: "kube-api-access-xjnhs") pod "31db999d-78c1-4635-9d71-a17f4a4a4782" (UID: "31db999d-78c1-4635-9d71-a17f4a4a4782"). InnerVolumeSpecName "kube-api-access-xjnhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:11:01.662258 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.662238 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31db999d-78c1-4635-9d71-a17f4a4a4782-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31db999d-78c1-4635-9d71-a17f4a4a4782" (UID: "31db999d-78c1-4635-9d71-a17f4a4a4782"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:11:01.662321 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.662242 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-dshm" (OuterVolumeSpecName: "dshm") pod "31db999d-78c1-4635-9d71-a17f4a4a4782" (UID: "31db999d-78c1-4635-9d71-a17f4a4a4782"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:11:01.761003 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.760979 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-tmp-dir\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:11:01.761003 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.761001 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31db999d-78c1-4635-9d71-a17f4a4a4782-tls-certs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:11:01.761003 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.761009 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31db999d-78c1-4635-9d71-a17f4a4a4782-dshm\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:11:01.761194 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.761017 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjnhs\" (UniqueName: \"kubernetes.io/projected/31db999d-78c1-4635-9d71-a17f4a4a4782-kube-api-access-xjnhs\") on node \"ip-10-0-131-194.ec2.internal\" DevicePath \"\"" Sep 16 01:11:01.893145 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.893079 2567 generic.go:356] "Generic (PLEG): container finished" podID="31db999d-78c1-4635-9d71-a17f4a4a4782" containerID="30d17d814f5cce29ee1df9781ec9bad86cac1f14d18e8ef0724c79f5c08955a0" exitCode=0 Sep 16 01:11:01.893145 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.893122 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" Sep 16 01:11:01.893275 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.893122 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" event={"ID":"31db999d-78c1-4635-9d71-a17f4a4a4782","Type":"ContainerDied","Data":"30d17d814f5cce29ee1df9781ec9bad86cac1f14d18e8ef0724c79f5c08955a0"} Sep 16 01:11:01.893275 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.893252 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q" event={"ID":"31db999d-78c1-4635-9d71-a17f4a4a4782","Type":"ContainerDied","Data":"3400985ef6f14a6876f75e48c1201dc2a41e4e205f332dde23c4380c0194a70d"} Sep 16 01:11:01.893342 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.893276 2567 scope.go:117] "RemoveContainer" containerID="30d17d814f5cce29ee1df9781ec9bad86cac1f14d18e8ef0724c79f5c08955a0" Sep 16 01:11:01.903102 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.903084 2567 scope.go:117] "RemoveContainer" containerID="30d17d814f5cce29ee1df9781ec9bad86cac1f14d18e8ef0724c79f5c08955a0" Sep 16 01:11:01.903391 ip-10-0-131-194 kubenswrapper[2567]: E0916 01:11:01.903374 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d17d814f5cce29ee1df9781ec9bad86cac1f14d18e8ef0724c79f5c08955a0\": container with ID starting with 30d17d814f5cce29ee1df9781ec9bad86cac1f14d18e8ef0724c79f5c08955a0 not found: ID does not exist" containerID="30d17d814f5cce29ee1df9781ec9bad86cac1f14d18e8ef0724c79f5c08955a0" Sep 16 01:11:01.903441 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.903401 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d17d814f5cce29ee1df9781ec9bad86cac1f14d18e8ef0724c79f5c08955a0"} err="failed to get container status \"30d17d814f5cce29ee1df9781ec9bad86cac1f14d18e8ef0724c79f5c08955a0\": rpc error: code = NotFound desc = could not find container \"30d17d814f5cce29ee1df9781ec9bad86cac1f14d18e8ef0724c79f5c08955a0\": container with ID starting with 30d17d814f5cce29ee1df9781ec9bad86cac1f14d18e8ef0724c79f5c08955a0 not found: ID does not exist" Sep 16 01:11:01.914469 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.914440 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q"] Sep 16 01:11:01.918351 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:01.918322 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-0a42620d-kserve-5b9f48687-ft68q"] Sep 16 01:11:02.118055 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:02.118016 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31db999d-78c1-4635-9d71-a17f4a4a4782" path="/var/lib/kubelet/pods/31db999d-78c1-4635-9d71-a17f4a4a4782/volumes" Sep 16 01:11:08.843412 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:08.843370 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:11:27.772095 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:27.772055 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:11:27.829384 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:11:27.829352 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:12:10.670794 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:12:10.670765 2567 scope.go:117] "RemoveContainer" containerID="5114413156a514653bd9badcef76a54626cbf6effdf02e2b9190efa4e87337b8" Sep 16 01:12:42.664239 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:12:42.664205 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:12:42.720933 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:12:42.720904 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:14:14.542866 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:14.542828 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:14:21.438667 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:21.438628 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:14:27.637768 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:27.637730 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:14:34.938746 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:34.938664 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jlmxq"] Sep 16 01:14:37.750290 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:37.750255 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:14:37.814241 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:37.814203 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" limit=1024 Sep 16 01:14:41.003487 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:41.003450 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fpm24_be34ccf1-b7da-47ab-9134-2813a37e07a1/global-pull-secret-syncer/0.log" Sep 16 01:14:41.111508 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:41.111476 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jzl25_1cd8f1ba-a180-412d-ac9b-8853de8af7ae/konnectivity-agent/0.log" Sep 16 01:14:41.207180 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:41.207148 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-194.ec2.internal_67a2beadf904429fdc039f04a692727f/haproxy/0.log" Sep 16 01:14:46.203387 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:46.203345 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jlmxq_9f037143-52b9-4170-a3da-eccf4e9c2aa9/limitador/0.log" Sep 16 01:14:47.807269 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:47.807236 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-sldj9_2d4219f4-4ffc-4733-b732-eb264c641245/kube-state-metrics/0.log" Sep 16 01:14:47.824114 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:47.824081 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-sldj9_2d4219f4-4ffc-4733-b732-eb264c641245/kube-rbac-proxy-main/0.log" Sep 16 01:14:47.849507 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:47.849485 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-sldj9_2d4219f4-4ffc-4733-b732-eb264c641245/kube-rbac-proxy-self/0.log" Sep 16 01:14:48.018229 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:48.018202 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h8w5x_ca98b25c-bcff-40b9-8182-15f72dde4b26/node-exporter/0.log" Sep 16 01:14:48.035533 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:48.035505 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h8w5x_ca98b25c-bcff-40b9-8182-15f72dde4b26/kube-rbac-proxy/0.log" Sep 16 01:14:48.063814 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:48.063757 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h8w5x_ca98b25c-bcff-40b9-8182-15f72dde4b26/init-textfile/0.log" Sep 16 01:14:48.193905 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:48.193883 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-xsrqz_8746e8b6-9c83-440d-a0f9-4b9070ef6104/kube-rbac-proxy-main/0.log" Sep 16 01:14:48.217721 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:48.217697 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-xsrqz_8746e8b6-9c83-440d-a0f9-4b9070ef6104/kube-rbac-proxy-self/0.log" Sep 16 01:14:48.241136 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:48.241113 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-xsrqz_8746e8b6-9c83-440d-a0f9-4b9070ef6104/openshift-state-metrics/0.log" Sep 16 01:14:48.453812 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:48.453734 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-76pzb_687690d1-dc38-41a7-88a5-ebbb24b3f30d/prometheus-operator/0.log" Sep 16 01:14:48.470793 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:48.470770 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-76pzb_687690d1-dc38-41a7-88a5-ebbb24b3f30d/kube-rbac-proxy/0.log" Sep 16 01:14:48.534724 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:48.534703 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b7ffbcf74-jr2c6_128bc9a9-1408-401b-a216-64f8a7197189/telemeter-client/0.log" Sep 16 01:14:48.555781 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:48.555759 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b7ffbcf74-jr2c6_128bc9a9-1408-401b-a216-64f8a7197189/reload/0.log" Sep 16 01:14:48.576392 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:48.576367 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b7ffbcf74-jr2c6_128bc9a9-1408-401b-a216-64f8a7197189/kube-rbac-proxy/0.log" Sep 16 01:14:50.994424 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:50.994393 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs"] Sep 16 01:14:50.994777 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:50.994768 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31db999d-78c1-4635-9d71-a17f4a4a4782" containerName="main" Sep 16 01:14:50.994824 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:50.994781 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="31db999d-78c1-4635-9d71-a17f4a4a4782" containerName="main" Sep 16 01:14:50.994860 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:50.994851 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f89837a-e1cf-4d80-a25c-145602b37ca7" containerName="istio-proxy" Sep 16 01:14:50.994894 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:50.994860 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="31db999d-78c1-4635-9d71-a17f4a4a4782" containerName="main" Sep 16 01:14:50.997988 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:50.997972 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" Sep 16 01:14:51.001191 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.001169 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jd2dw\"/\"default-dockercfg-v76dc\"" Sep 16 01:14:51.002659 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.002627 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jd2dw\"/\"kube-root-ca.crt\"" Sep 16 01:14:51.002777 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.002660 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jd2dw\"/\"openshift-service-ca.crt\"" Sep 16 01:14:51.008615 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.008591 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs"] Sep 16 01:14:51.086215 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.086186 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f02f34d-b4a7-4b61-9224-a7214eda446d-lib-modules\") pod \"perf-node-gather-daemonset-22tqs\" (UID: \"2f02f34d-b4a7-4b61-9224-a7214eda446d\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" Sep 16 01:14:51.086336 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.086219 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2f02f34d-b4a7-4b61-9224-a7214eda446d-podres\") pod \"perf-node-gather-daemonset-22tqs\" (UID: \"2f02f34d-b4a7-4b61-9224-a7214eda446d\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" Sep 16 01:14:51.086336 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.086240 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhrmj\" (UniqueName: \"kubernetes.io/projected/2f02f34d-b4a7-4b61-9224-a7214eda446d-kube-api-access-rhrmj\") pod \"perf-node-gather-daemonset-22tqs\" (UID: \"2f02f34d-b4a7-4b61-9224-a7214eda446d\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" Sep 16 01:14:51.086336 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.086286 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f02f34d-b4a7-4b61-9224-a7214eda446d-sys\") pod \"perf-node-gather-daemonset-22tqs\" (UID: \"2f02f34d-b4a7-4b61-9224-a7214eda446d\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" Sep 16 01:14:51.086336 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.086309 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2f02f34d-b4a7-4b61-9224-a7214eda446d-proc\") pod \"perf-node-gather-daemonset-22tqs\" (UID: \"2f02f34d-b4a7-4b61-9224-a7214eda446d\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" Sep 16 01:14:51.187389 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.187360 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f02f34d-b4a7-4b61-9224-a7214eda446d-lib-modules\") pod \"perf-node-gather-daemonset-22tqs\" (UID: \"2f02f34d-b4a7-4b61-9224-a7214eda446d\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" Sep 16 01:14:51.187495 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.187393 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2f02f34d-b4a7-4b61-9224-a7214eda446d-podres\") pod \"perf-node-gather-daemonset-22tqs\" (UID: \"2f02f34d-b4a7-4b61-9224-a7214eda446d\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" Sep 16 01:14:51.187495 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.187412 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhrmj\" (UniqueName: \"kubernetes.io/projected/2f02f34d-b4a7-4b61-9224-a7214eda446d-kube-api-access-rhrmj\") pod \"perf-node-gather-daemonset-22tqs\" (UID: \"2f02f34d-b4a7-4b61-9224-a7214eda446d\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" Sep 16 01:14:51.187614 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.187486 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f02f34d-b4a7-4b61-9224-a7214eda446d-sys\") pod \"perf-node-gather-daemonset-22tqs\" (UID: \"2f02f34d-b4a7-4b61-9224-a7214eda446d\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" Sep 16 01:14:51.187614 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.187528 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2f02f34d-b4a7-4b61-9224-a7214eda446d-proc\") pod \"perf-node-gather-daemonset-22tqs\" (UID: \"2f02f34d-b4a7-4b61-9224-a7214eda446d\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" Sep 16 01:14:51.187614 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.187537 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f02f34d-b4a7-4b61-9224-a7214eda446d-lib-modules\") pod \"perf-node-gather-daemonset-22tqs\" (UID: \"2f02f34d-b4a7-4b61-9224-a7214eda446d\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" Sep 16 01:14:51.187614 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.187539 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2f02f34d-b4a7-4b61-9224-a7214eda446d-podres\") pod \"perf-node-gather-daemonset-22tqs\" (UID: \"2f02f34d-b4a7-4b61-9224-a7214eda446d\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" Sep 16 01:14:51.187614 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.187578 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f02f34d-b4a7-4b61-9224-a7214eda446d-sys\") pod \"perf-node-gather-daemonset-22tqs\" (UID: \"2f02f34d-b4a7-4b61-9224-a7214eda446d\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" Sep 16 01:14:51.187802 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.187636 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2f02f34d-b4a7-4b61-9224-a7214eda446d-proc\") pod \"perf-node-gather-daemonset-22tqs\" (UID: \"2f02f34d-b4a7-4b61-9224-a7214eda446d\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" Sep 16 01:14:51.195725 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.195708 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhrmj\" (UniqueName: \"kubernetes.io/projected/2f02f34d-b4a7-4b61-9224-a7214eda446d-kube-api-access-rhrmj\") pod \"perf-node-gather-daemonset-22tqs\" (UID: \"2f02f34d-b4a7-4b61-9224-a7214eda446d\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" Sep 16 01:14:51.309195 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.309175 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" Sep 16 01:14:51.579936 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:51.579913 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs"] Sep 16 01:14:51.581880 ip-10-0-131-194 kubenswrapper[2567]: W0916 01:14:51.581842 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2f02f34d_b4a7_4b61_9224_a7214eda446d.slice/crio-eaf147d86380133c994e87d527a5e7fbd11d952b5f4d0f181e2cf7f87e6dbbf9 WatchSource:0}: Error finding container eaf147d86380133c994e87d527a5e7fbd11d952b5f4d0f181e2cf7f87e6dbbf9: Status 404 returned error can't find the container with id eaf147d86380133c994e87d527a5e7fbd11d952b5f4d0f181e2cf7f87e6dbbf9 Sep 16 01:14:52.049902 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:52.049871 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" event={"ID":"2f02f34d-b4a7-4b61-9224-a7214eda446d","Type":"ContainerStarted","Data":"1c899e6eda4bce3209b36fb66245b1a930468a2a12049307fb09a9e123c2a63a"} Sep 16 01:14:52.050272 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:52.049932 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" event={"ID":"2f02f34d-b4a7-4b61-9224-a7214eda446d","Type":"ContainerStarted","Data":"eaf147d86380133c994e87d527a5e7fbd11d952b5f4d0f181e2cf7f87e6dbbf9"} Sep 16 01:14:52.050272 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:52.049982 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" Sep 16 01:14:52.067830 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:52.067783 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" podStartSLOduration=2.067768737 podStartE2EDuration="2.067768737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:14:52.065622471 +0000 UTC m=+2442.427846003" watchObservedRunningTime="2026-09-16 01:14:52.067768737 +0000 UTC m=+2442.429992270" Sep 16 01:14:53.342459 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:53.342427 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w2wk7_6eeff833-d3c0-4fff-b06a-f9870ba630de/dns/0.log" Sep 16 01:14:53.361028 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:53.361004 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w2wk7_6eeff833-d3c0-4fff-b06a-f9870ba630de/kube-rbac-proxy/0.log" Sep 16 01:14:53.381227 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:53.381200 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cd4mj_abe1a6aa-c150-47db-9b2f-e26f6415570f/dns-node-resolver/0.log" Sep 16 01:14:53.985113 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:53.985082 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-79b88569f4-xrrcb_2af5f40b-8530-42d2-b0d6-8c15f74169df/registry/0.log" Sep 16 01:14:54.030632 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:54.030603 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q86vf_d59da895-813d-4240-848d-9513b5886d1c/node-ca/0.log" Sep 16 01:14:54.967780 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:54.967753 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8_3c87bb77-2bef-4b57-80f0-b77131f2425d/istio-proxy/0.log" Sep 16 01:14:55.620766 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:55.620737 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-kt8vk_8acf6a17-0628-4c35-bf78-c3dc03adecd8/discovery/0.log" Sep 16 01:14:56.673599 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:56.673568 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bspl6_2465e5a4-b67b-42f6-9934-f65c4c2fadb3/serve-healthcheck-canary/0.log" Sep 16 01:14:57.251864 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:57.251837 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jhgj5_b1596382-2cf4-4b0a-83cd-8e6c2ff16981/kube-rbac-proxy/0.log" Sep 16 01:14:57.272463 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:57.272439 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jhgj5_b1596382-2cf4-4b0a-83cd-8e6c2ff16981/exporter/0.log" Sep 16 01:14:57.292947 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:57.292923 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jhgj5_b1596382-2cf4-4b0a-83cd-8e6c2ff16981/extractor/0.log" Sep 16 01:14:58.064914 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:58.064883 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-22tqs" Sep 16 01:14:59.902202 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:14:59.902169 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-d77jm_074346fa-1298-4edd-b974-baf07217275a/server/0.log" Sep 16 01:15:00.035411 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:00.035379 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lgdjj_0a62d9e3-188b-41ef-96fa-47ac89811ce9/manager/0.log" Sep 16 01:15:00.130421 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:00.130383 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825355-jc2bz"] Sep 16 01:15:00.134346 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:00.134311 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825355-jc2bz" Sep 16 01:15:00.136964 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:00.136933 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-npltg\"" Sep 16 01:15:00.141792 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:00.141767 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825355-jc2bz"] Sep 16 01:15:00.147788 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:00.147767 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825355-jc2bz" Sep 16 01:15:00.159794 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:00.159773 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55d645f7ff-gv42q_0d37134c-6aa4-4876-8992-0274c84ebd5c/manager/0.log" Sep 16 01:15:00.176882 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:00.176854 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55d645f7ff-gv42q_0d37134c-6aa4-4876-8992-0274c84ebd5c/copy-manifests/0.log" Sep 16 01:15:00.422482 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:00.422453 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825355-jc2bz"] Sep 16 01:15:00.423859 ip-10-0-131-194 kubenswrapper[2567]: W0916 01:15:00.423832 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce669630_7dc9_4bcc_9946_0d7ad378665e.slice/crio-9893993a7e1f390f9339a604aaffa377d7241648e8bc490443ceef26a83aed92 WatchSource:0}: Error finding container 9893993a7e1f390f9339a604aaffa377d7241648e8bc490443ceef26a83aed92: Status 404 returned error can't find the container with id 9893993a7e1f390f9339a604aaffa377d7241648e8bc490443ceef26a83aed92 Sep 16 01:15:01.098323 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:01.098247 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825355-jc2bz" event={"ID":"ce669630-7dc9-4bcc-9946-0d7ad378665e","Type":"ContainerStarted","Data":"c063891c8c95b0080f2c913a63170ec74dcdfc4ea0e4c59014f6f01581fc41d9"} Sep 16 01:15:01.098323 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:01.098303 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825355-jc2bz" event={"ID":"ce669630-7dc9-4bcc-9946-0d7ad378665e","Type":"ContainerStarted","Data":"9893993a7e1f390f9339a604aaffa377d7241648e8bc490443ceef26a83aed92"} Sep 16 01:15:01.115224 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:01.115176 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825355-jc2bz" podStartSLOduration=1.115161137 podStartE2EDuration="1.115161137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:15:01.113027425 +0000 UTC m=+2451.475250980" watchObservedRunningTime="2026-09-16 01:15:01.115161137 +0000 UTC m=+2451.477384669" Sep 16 01:15:01.928269 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:01.928229 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-864798458c-v8zs5_5387ab5d-9440-4c3c-88ea-3d2188cbdbed/manager/0.log" Sep 16 01:15:01.953358 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:01.953335 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-cgw99_5658bcbf-21df-411e-b218-43a8ec27c0d4/openshift-lws-operator/0.log" Sep 16 01:15:01.977647 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:01.977622 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8_3c87bb77-2bef-4b57-80f0-b77131f2425d/istio-proxy/0.log" Sep 16 01:15:02.106687 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:02.106652 2567 generic.go:356] "Generic (PLEG): container finished" podID="ce669630-7dc9-4bcc-9946-0d7ad378665e" containerID="c063891c8c95b0080f2c913a63170ec74dcdfc4ea0e4c59014f6f01581fc41d9" exitCode=0 Sep 16 01:15:02.107281 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:02.106782 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825355-jc2bz" event={"ID":"ce669630-7dc9-4bcc-9946-0d7ad378665e","Type":"ContainerDied","Data":"c063891c8c95b0080f2c913a63170ec74dcdfc4ea0e4c59014f6f01581fc41d9"} Sep 16 01:15:02.637007 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:02.636976 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-kt8vk_8acf6a17-0628-4c35-bf78-c3dc03adecd8/discovery/0.log" Sep 16 01:15:03.237425 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:03.237401 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825355-jc2bz" Sep 16 01:15:04.118015 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:04.117989 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825355-jc2bz" Sep 16 01:15:04.118206 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:04.118076 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825355-jc2bz" event={"ID":"ce669630-7dc9-4bcc-9946-0d7ad378665e","Type":"ContainerDied","Data":"9893993a7e1f390f9339a604aaffa377d7241648e8bc490443ceef26a83aed92"} Sep 16 01:15:04.118206 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:04.118125 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9893993a7e1f390f9339a604aaffa377d7241648e8bc490443ceef26a83aed92" Sep 16 01:15:04.272685 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:04.272646 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-4vlh8_3c87bb77-2bef-4b57-80f0-b77131f2425d/istio-proxy/0.log" Sep 16 01:15:04.937449 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:04.937421 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-kt8vk_8acf6a17-0628-4c35-bf78-c3dc03adecd8/discovery/0.log" Sep 16 01:15:08.701427 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:08.701397 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9spsl_8105e12c-26da-4f67-a85b-734fb527ad2b/kube-multus-additional-cni-plugins/0.log" Sep 16 01:15:08.721138 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:08.721111 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9spsl_8105e12c-26da-4f67-a85b-734fb527ad2b/egress-router-binary-copy/0.log" Sep 16 01:15:08.740845 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:08.740819 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9spsl_8105e12c-26da-4f67-a85b-734fb527ad2b/cni-plugins/0.log" Sep 16 01:15:08.762985 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:08.762956 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9spsl_8105e12c-26da-4f67-a85b-734fb527ad2b/bond-cni-plugin/0.log" Sep 16 01:15:08.783590 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:08.783565 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9spsl_8105e12c-26da-4f67-a85b-734fb527ad2b/routeoverride-cni/0.log" Sep 16 01:15:08.805719 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:08.805698 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9spsl_8105e12c-26da-4f67-a85b-734fb527ad2b/whereabouts-cni-bincopy/0.log" Sep 16 01:15:08.828396 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:08.828369 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9spsl_8105e12c-26da-4f67-a85b-734fb527ad2b/whereabouts-cni/0.log" Sep 16 01:15:08.891120 ip-10-0-131-194 kubenswrapper[2567]: I0916 01:15:08.891095 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bglnh_d5885041-68cc-4ec8-85e9-9555d7baa6c9/kube-multus/0.log"